https://www.vistrails.org//api.php?action=feedcontributions&user=Remi&feedformat=atomVistrailsWiki - User contributions [en]2024-03-29T13:51:06ZUser contributionsMediaWiki 1.36.2https://www.vistrails.org//index.php?title=Main_Page&diff=22198Main Page2021-12-13T15:50:55Z<p>Remi: </p>
<hr />
<div>This is the official website for VisTrails, an open-source scientific workflow and provenance management system that supports data exploration and visualization.<br />
<br />
VisTrails provides a general infrastructure that can be combined with existing systems and libraries. Official releases come with several packages, including VTK, matplotlib, and ImageMagick. The system is written in Python/Qt and runs on Mac, Unix and Windows.<br />
<br />
[[#Downloading VisTrails|Read more >]]<br />
<br />
'''Please note that as of 2018, we are no longer maintaining VisTrails.'''<br />
<br />
== Downloading VisTrails ==<br />
Current and past releases are available on our [[downloads|download page]].<br />
<br />
== News ==<br />
* Unfortunately we can no longer maintain VisTrails<br />
* [2016-05-03] VisTrails 2.2.4 has been released and is available on the [[Downloads]] page.<br />
* [2015-09-11] SAHM VisTrails training course at USGS: http://pubs.usgs.gov/fs/2014/3007/pdf/fs2014-3007.pdf<br />
* [2015-07-10] VisTrails 2.2.2 has been released and is available on the [[Downloads]] page.<br />
* [2015-04-22] VisTrails 2.2 has been released and is available on the [[Downloads]] page.<br />
* [2015-03-21] VisTrails 2.1.5 has been released and is available on the [[Downloads]] page.<br />
* [2014-08-25] VisTrails 2.1.4 has been released and is available on the [[Downloads]] page.<br />
* VisTrails 2.0.3 has been released and is available on the [[Downloads]] page.<br />
* To learn more about how to explore climate data using DV3D and ParaView in UV-CDAT, check out http://www.kitware.com/source/home/post/105<br />
* Comparison of two forecasts of Hurricane Sandy using the DV3D slicer in UV-CDAT http://www.youtube.com/watch?v=BFN0RzN1hSE&feature=em-share_video_user<br />
* Presentation on the use of VisTrails at NASA: VisTrails: Workflow Management System on NEX - https://c3.nasa.gov/nex/static/media/other/Vistrails-Votava.pdf<br />
* VisTrails 2.0 has been released and it is available for download on our [[Downloads]] page. VisTrails is now distributed under the BSD license and its GUI was completely re-designed to make it easier to manage and interact with multiple workflows. It also includes a much improved [http://panda.poly.edu:8080/usersguide Users' Manual]. See the [http://sourceforge.net/projects/vistrails/files/vistrails/v2.0/README/view release notes] for a complete list of changes. Some notable features include: <br />
** the ability to create mashups; <br />
** support for the creation of reproducible papers; <br />
** a provenance viewer; <br />
** support for visual diffs as well as queries over workflows in different vistrails; <br />
** a Control Flow Assistant to simplify the creation of workflows with loops and conditional constructs;<br />
** support for global variables, so that modules deriving from Constant can be used to create variables that are shared among multiple workflows. <br />
<br />
[[Image:Screen Shot 2012-01-12 at 2.50.19 PM.png|200px|right]]<br />
* UV-CDAT 1.0.0 has been released. More detailed information and binaries are available at http://uvcdat.llnl.gov<br />
** UV-CDAT was demoed at [http://ipaw2012.bren.ucsb.edu IPAW 2012]<br />
** Video tutorials are available at http://uv-cdat.llnl.gov/wiki/VideoTutorials<br />
** Binaries (Mac/RedHAT6/Ubuntu) or build-from-sources options can be found here: http://uv-cdat.llnl.gov/install/<br />
** Please report any issues you encounter via bugzilla at: http://uv-cdat.llnl.gov/bugzilla/<br />
<br />
<br />
* [[Older News]]<br />
<br />
== VisTrails Overview ==<br />
[[Image:Cosmology_example.png|480px|right]]<br />
VisTrails is an open-source scientific workflow and provenance management system that provides support for simulations, data exploration and visualization. Whereas workflows have been traditionally used to automate repetitive tasks, for applications that are exploratory in nature, such as simulations, data analysis and visualization, very little is repeated---change is the norm. As an engineer or scientist generates and evaluates hypotheses about data under study, a series of different, albeit related, workflows are created while a workflow is adjusted in an interactive process. VisTrails was designed to manage these rapidly-evolving workflows. <br />
<br />
A key distinguishing feature of VisTrails is a comprehensive provenance infrastructure that maintains detailed history information about the steps followed and data derived in the course of an exploratory task: VisTrails maintains provenance of data products, of the workflows that derive these products and their executions. This information is persisted as XML files or in a relational database, and it allows users to navigate workflow versions in an intuitive way, to undo changes but not lose any results, to visually compare different workflows and their results, and to examine the actions that led to a result. It also enables a series operations and user interfaces that simplify workflow design and use, including the ability to create and refine workflows by analogy and to query workflows by example.<br />
<br />
VisTrails supports the creation and execution of workflows. It allows the combination of loosely-coupled resources, specialized libraries, grid and Web services. The released version comes with support for several packages including, [http://www.vtk.org/ VTK], [http://www.imagemagick.org/script/index.php Image Magick], Web Services, and [http://www.scipy.org/PyLab pylab]. You can also download [[UserContributedPackages|packages contributed by our users]], as well as [http://panda.poly.edu:8080/usersguide/v2.1/html/packages.html create your own packages/modules]. There are also a number of [[Projects using VisTrails|projects extending or building on top of VisTrails]].<br />
Workflows can be run interactively, through the VisTrails GUI, or in batch using a VisTrails server.<br />
<br />
VisTrails 1.0 was released in October 2007. Since then, the system has been downloaded thousands of times. VisTrails is written in Python and it uses the multi-platform Qt library for its user interface. It runs on Mac, Linux and Windows. You can download VisTrails from http://panda.poly.edu:8080/index.php/Downloads<br />
<br />
=== Publishing Reproducible Results ===<br />
VisTrails 2.0 allows the inclusion of reproducible results in LaTeX/PDF documents. We provide a LaTeX package that allows users to add links to their results in the LaTeX source. For example: <br />
<br />
<code><br />
\usepackage{vistrails}<br />
<br />
...<br />
<br />
\begin{figure}<br />
<br />
\begin{center}<br />
<br />
\subfigure[a=0.9]{\vistrail[filename=alps.vt, version=2, pdf]{width=8cm}}<br />
<br />
\subfigure[a=0.9]{\vistrail[filename=alps.vt, version=11, pdf,buildalways]{width=8cm}}<br />
<br />
\caption{A figure produced by an ALPS VisTrails workflow. Clicking the figure retrieves the workflow used to <br />
create it. Opening that workflow on a machine with VisTrails and ALPS installed lets the reader execute the full calculation.}<br />
<br />
\end{center}<br />
<br />
\end{figure}<br />
</code><br />
<br />
Once the LaTeX document is compiled, the figure in the PDF becomes active, and when clicked, it will invoke VisTrails and reproduce the result.<br />
You can also upload your results to [http://www.crowdlabs.org CrowdLabs] and export them to Web sites or Wikis, where users can interact with them through a Web browser. See e.g., http://www.crowdlabs.org/vistrails/medleys/details/26/<br />
<br />
=== Building Workflows ===<br />
VisTrails supports workflows expressed as dataflows. It can also support functional loops and conditional statements (new in VisTrails 1.3!). In addition, VisTrails is easily extensible; users can quickly create new modules or wrap existing code or libraries for use as modules in VisTrails. Also, the <code>PythonSource</code> module can be used to write one-time translations or test code in a workflow. See the [http://sourceforge.net/projects/vistrails/files/vistrails/vistrails-usersguide-1.3-rev198.pdf/download Users' Guide] for more details on all of these features.<br />
<br />
=== Combining VisTrails Provenance with other Tools ===<br />
[[Image:DisplayWall_ipod_screenshot.png|180px|right]]<br />
The VisTrails provenance infrastructure can also be combined with interactive tools. We are developing a series of provenance plugins for different tools, including [https://wci.llnl.gov/codes/visit VisIt] and [http://www.paraview.org/ ParaView]. The [http://panda.poly.edu:8080/index.php/ParaView_Plugin open-source ParaView plugin has been released].<br />
<br />
=== Visualizing and Comparing Workflow Results ===<br />
VisTrails provides a ''spreadsheet'' where users can compare the results of multiple workflows, or multiple workflow runs. Besides using a desktop, you can display the results of VisTrails workflows on a display wall or on an iPod. Users can also compare different workflows using the visual difference interface.<br />
<br />
=== Querying and Refining Workflows by Example ===<br />
Users can quickly construct expressive queries over a collection of workflows by example, using the same familiar interface they use to build workflows. VisTrails also provides an analogy mechanism whereby users can perform complex modifications to workflows by analogy, without having to directly modify the workflow specifications.<br />
<br />
==[[Documentation|System Documentation]]==<br />
If you want to get more information about VisTrails you can consult the Users' Manual or the FAQ (see links below). If you have questions that are not answered in these resources, you can post it to the users' mailing list: http://panda.poly.edu:8080/index.php/MailingLists<br />
<br />
To report bugs, please use our bug tracker: https://github.com/VisTrails/VisTrails/issues<br />
<br />
=== [[Documentation#Screen Shots|Screen Shots]]===<br />
=== [[Users_Guide|User's Guide]]===<br />
A Web version of the Users Guide is currently under development and can be accessed at http://panda.poly.edu:8080/usersguide.<br />
=== [[FAQ]]===<br />
<br />
==[[MailingLists|Mailing Lists]]==<br />
==[[Publications, Tutorials and Presentations]]==<br />
==[[People]]==<br />
<br />
== [http://panda.poly.edu:8080/index.php/Projects_using_VisTrails Some Projects using VisTrails] ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! USGS Habitat modeling<br />
|-<br />
| [[Image:usgs.png|200px|left]]<br />
|<br />
|}<br />
<br />
{| class="wikitable"<br />
|-<br />
! NASA Climate Data Analysis <br />
! DOE CDAT<br />
! ALPS Simulations<br />
|-<br />
| [[Image:nasa.png|200px|left]]<br />
| [[Image:cdat.png|200px|left]]<br />
| [[Image:alps-shot.png|200px|left]]<br />
|<br />
|}<br />
<br />
{| class="wikitable"<br />
|-<br />
! NSF STC CMOP<br />
! NSF CDI Wildfire<br />
! NSF DataONE-EVA<br />
|-<br />
| [[Image:cmop-ss.png|200px|left]]<br />
| [[Image:wildfire.png|200px|center]]<br />
| [[Image:eva.png|200px|left]]<br />
|<br />
|}<br />
[http://panda.poly.edu:8080/index.php/Projects_using_VisTrails See other projects]<br />
<br />
== [[Vistrails and Teaching]] ==<br />
<br />
The VisTrails system has been used a tool to teach different courses, including Scientific Visualization and Digital Media.<br />
For more information on the educational uses of VisTrails see [http://panda.poly.edu:8080/index.php/Vistrails_and_Teaching VisTrails and Teaching]. Our [http://www.cs.utah.edu/~juliana/pub/vistrails-teaching-eurographics2010.pdf paper] describing a provenance-rich teaching methodology received the Best Paper Award at the Eurographics 2010 Education program.<br />
<br />
== Sponsors ==<br />
<br />
This work has been partially supported by the National Science Foundation under grants <br />
[http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0905385 IIS-0905385],<br />
[http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0844572 IIS-0844572],<br />
[http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0746500 IIS CAREER-0746500],<br />
[http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0751152 CNS-0751152],<br />
[http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0513692 IIS-0513692], <br />
[http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0401498 CCF-0401498], <br />
[http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0541560 CNS-0541560], [http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0405402 OISE-0405402], <br />
[http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0424602 OCE-0424602], <br />
[http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0524096 CNS-0524096],<br />
[http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0534628 IIS-0534628],<br />
the Department of Energy under the SciDAC program<br />
([http://www.scidac.org/compsci/SDM.html SDM]<br />
and<br />
[http://www.scidac.org/viz/VACET.html VACET]) and UV-CDAT,<br />
IBM Faculty Awards (2005, 2006, 2007, and 2008) and a University of Utah Seed Grant.<br />
<br />
== Related Sites ==<br />
<br />
[[BirdVis]]<br />
<br />
[http://www.crowdlabs.org CrowdLabs]<br />
<br />
[[RepeatabilityCentral]]<br />
<br />
[[ProvenanceAnalytics]]<br />
<br />
[[Provenance: potpourri]]</div>Remihttps://www.vistrails.org//index.php?title=Main_Page&diff=18853Main Page2019-09-19T15:38:22Z<p>Remi: /* News */ No longer maintained</p>
<hr />
<div>This is the official website for VisTrails, an open-source scientific workflow and provenance management system that supports data exploration and visualization.<br />
<br />
VisTrails provides a general infrastructure that can be combined with existing systems and libraries. Official releases come with several packages, including VTK, matplotlib, and ImageMagick. The system is written in Python/Qt and runs on Mac, Unix and Windows.<br />
<br />
[[#Downloading VisTrails|Read more >]]<br />
<br />
'''Please note that at this time, we are no longer maintaining VisTrails.'''<br />
<br />
== Downloading VisTrails ==<br />
Current and past releases are available on our [[downloads|download page]].<br />
<br />
== News ==<br />
* Unfortunately we can no longer maintain VisTrails<br />
* [2016-05-03] VisTrails 2.2.4 has been released and is available on the [[Downloads]] page.<br />
* [2015-09-11] SAHM VisTrails training course at USGS: http://pubs.usgs.gov/fs/2014/3007/pdf/fs2014-3007.pdf<br />
* [2015-07-10] VisTrails 2.2.2 has been released and is available on the [[Downloads]] page.<br />
* [2015-04-22] VisTrails 2.2 has been released and is available on the [[Downloads]] page.<br />
* [2015-03-21] VisTrails 2.1.5 has been released and is available on the [[Downloads]] page.<br />
* [2014-08-25] VisTrails 2.1.4 has been released and is available on the [[Downloads]] page.<br />
* VisTrails 2.0.3 has been released and is available on the [[Downloads]] page.<br />
* To learn more about how to explore climate data using DV3D and ParaView in UV-CDAT, check out http://www.kitware.com/source/home/post/105<br />
* Comparison of two forecasts of Hurricane Sandy using the DV3D slicer in UV-CDAT http://www.youtube.com/watch?v=BFN0RzN1hSE&feature=em-share_video_user<br />
* Presentation on the use of VisTrails at NASA: VisTrails: Workflow Management System on NEX - https://c3.nasa.gov/nex/static/media/other/Vistrails-Votava.pdf<br />
* VisTrails 2.0 has been released and it is available for download on our [[Downloads]] page. VisTrails is now distributed under the BSD license and its GUI was completely re-designed to make it easier to manage and interact with multiple workflows. It also includes a much improved [http://www.vistrails.org/usersguide Users' Manual]. See the [http://sourceforge.net/projects/vistrails/files/vistrails/v2.0/README/view release notes] for a complete list of changes. Some notable features include: <br />
** the ability to create mashups; <br />
** support for the creation of reproducible papers; <br />
** a provenance viewer; <br />
** support for visual diffs as well as queries over workflows in different vistrails; <br />
** a Control Flow Assistant to simplify the creation of workflows with loops and conditional constructs;<br />
** support for global variables, so that modules deriving from Constant can be used to create variables that are shared among multiple workflows. <br />
<br />
[[Image:Screen Shot 2012-01-12 at 2.50.19 PM.png|200px|right]]<br />
* UV-CDAT 1.0.0 has been released. More detailed information and binaries are available at http://uvcdat.llnl.gov<br />
** UV-CDAT was demoed at [http://ipaw2012.bren.ucsb.edu IPAW 2012]<br />
** Video tutorials are available at http://uv-cdat.llnl.gov/wiki/VideoTutorials<br />
** Binaries (Mac/RedHAT6/Ubuntu) or build-from-sources options can be found here: http://uv-cdat.llnl.gov/install/<br />
** Please report any issues you encounter via bugzilla at: http://uv-cdat.llnl.gov/bugzilla/<br />
<br />
<br />
* [[Older News]]<br />
<br />
== VisTrails Overview ==<br />
[[Image:Cosmology_example.png|480px|right]]<br />
VisTrails is an open-source scientific workflow and provenance management system that provides support for simulations, data exploration and visualization. Whereas workflows have been traditionally used to automate repetitive tasks, for applications that are exploratory in nature, such as simulations, data analysis and visualization, very little is repeated---change is the norm. As an engineer or scientist generates and evaluates hypotheses about data under study, a series of different, albeit related, workflows are created while a workflow is adjusted in an interactive process. VisTrails was designed to manage these rapidly-evolving workflows. <br />
<br />
A key distinguishing feature of VisTrails is a comprehensive provenance infrastructure that maintains detailed history information about the steps followed and data derived in the course of an exploratory task: VisTrails maintains provenance of data products, of the workflows that derive these products and their executions. This information is persisted as XML files or in a relational database, and it allows users to navigate workflow versions in an intuitive way, to undo changes but not lose any results, to visually compare different workflows and their results, and to examine the actions that led to a result. It also enables a series operations and user interfaces that simplify workflow design and use, including the ability to create and refine workflows by analogy and to query workflows by example.<br />
<br />
VisTrails supports the creation and execution of workflows. It allows the combination of loosely-coupled resources, specialized libraries, grid and Web services. The released version comes with support for several packages including, [http://www.vtk.org/ VTK], [http://www.imagemagick.org/script/index.php Image Magick], Web Services, and [http://www.scipy.org/PyLab pylab]. You can also download [[UserContributedPackages|packages contributed by our users]], as well as [http://www.vistrails.org/usersguide/v2.1/html/packages.html create your own packages/modules]. There are also a number of [[Projects using VisTrails|projects extending or building on top of VisTrails]].<br />
Workflows can be run interactively, through the VisTrails GUI, or in batch using a VisTrails server.<br />
<br />
VisTrails 1.0 was released in October 2007. Since then, the system has been downloaded thousands of times. VisTrails is written in Python and it uses the multi-platform Qt library for its user interface. It runs on Mac, Linux and Windows. You can download VisTrails from http://www.vistrails.org/index.php/Downloads<br />
<br />
=== Publishing Reproducible Results ===<br />
VisTrails 2.0 allows the inclusion of reproducible results in LaTeX/PDF documents. We provide a LaTeX package that allows users to add links to their results in the LaTeX source. For example: <br />
<br />
<code><br />
\usepackage{vistrails}<br />
<br />
...<br />
<br />
\begin{figure}<br />
<br />
\begin{center}<br />
<br />
\subfigure[a=0.9]{\vistrail[filename=alps.vt, version=2, pdf]{width=8cm}}<br />
<br />
\subfigure[a=0.9]{\vistrail[filename=alps.vt, version=11, pdf,buildalways]{width=8cm}}<br />
<br />
\caption{A figure produced by an ALPS VisTrails workflow. Clicking the figure retrieves the workflow used to <br />
create it. Opening that workflow on a machine with VisTrails and ALPS installed lets the reader execute the full calculation.}<br />
<br />
\end{center}<br />
<br />
\end{figure}<br />
</code><br />
<br />
Once the LaTeX document is compiled, the figure in the PDF becomes active, and when clicked, it will invoke VisTrails and reproduce the result.<br />
You can also upload your results to [http://www.crowdlabs.org CrowdLabs] and export them to Web sites or Wikis, where users can interact with them through a Web browser. See e.g., http://www.crowdlabs.org/vistrails/medleys/details/26/<br />
<br />
=== Building Workflows ===<br />
VisTrails supports workflows expressed as dataflows. It can also support functional loops and conditional statements (new in VisTrails 1.3!). In addition, VisTrails is easily extensible; users can quickly create new modules or wrap existing code or libraries for use as modules in VisTrails. Also, the <code>PythonSource</code> module can be used to write one-time translations or test code in a workflow. See the [http://sourceforge.net/projects/vistrails/files/vistrails/vistrails-usersguide-1.3-rev198.pdf/download Users' Guide] for more details on all of these features.<br />
<br />
=== Combining VisTrails Provenance with other Tools ===<br />
[[Image:DisplayWall_ipod_screenshot.png|180px|right]]<br />
The VisTrails provenance infrastructure can also be combined with interactive tools. We are developing a series of provenance plugins for different tools, including [https://wci.llnl.gov/codes/visit VisIt] and [http://www.paraview.org/ ParaView]. The [http://www.vistrails.org/index.php/ParaView_Plugin open-source ParaView plugin has been released].<br />
<br />
=== Visualizing and Comparing Workflow Results ===<br />
VisTrails provides a ''spreadsheet'' where users can compare the results of multiple workflows, or multiple workflow runs. Besides using a desktop, you can display the results of VisTrails workflows on a display wall or on an iPod. Users can also compare different workflows using the visual difference interface.<br />
<br />
=== Querying and Refining Workflows by Example ===<br />
Users can quickly construct expressive queries over a collection of workflows by example, using the same familiar interface they use to build workflows. VisTrails also provides an analogy mechanism whereby users can perform complex modifications to workflows by analogy, without having to directly modify the workflow specifications.<br />
<br />
==[[Documentation|System Documentation]]==<br />
If you want to get more information about VisTrails you can consult the Users' Manual or the FAQ (see links below). If you have questions that are not answered in these resources, you can post it to the users' mailing list: http://www.vistrails.org/index.php/MailingLists<br />
<br />
To report bugs, please use our bug tracker: https://github.com/VisTrails/VisTrails/issues<br />
<br />
=== [[Documentation#Screen Shots|Screen Shots]]===<br />
=== [[Users_Guide|User's Guide]]===<br />
A Web version of the Users Guide is currently under development and can be accessed at http://www.vistrails.org/usersguide.<br />
=== [[FAQ]]===<br />
<br />
==[[MailingLists|Mailing Lists]]==<br />
==[[Publications, Tutorials and Presentations]]==<br />
==[[People]]==<br />
<br />
== [http://www.vistrails.org/index.php/Projects_using_VisTrails Some Projects using VisTrails] ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! USGS Habitat modeling<br />
|-<br />
| [[Image:usgs.png|200px|left]]<br />
|<br />
|}<br />
<br />
{| class="wikitable"<br />
|-<br />
! NASA Climate Data Analysis <br />
! DOE CDAT<br />
! ALPS Simulations<br />
|-<br />
| [[Image:nasa.png|200px|left]]<br />
| [[Image:cdat.png|200px|left]]<br />
| [[Image:alps-shot.png|200px|left]]<br />
|<br />
|}<br />
<br />
{| class="wikitable"<br />
|-<br />
! NSF STC CMOP<br />
! NSF CDI Wildfire<br />
! NSF DataONE-EVA<br />
|-<br />
| [[Image:cmop-ss.png|200px|left]]<br />
| [[Image:wildfire.png|200px|center]]<br />
| [[Image:eva.png|200px|left]]<br />
|<br />
|}<br />
[http://www.vistrails.org/index.php/Projects_using_VisTrails See other projects]<br />
<br />
== [[Vistrails and Teaching]] ==<br />
<br />
The VisTrails system has been used a tool to teach different courses, including Scientific Visualization and Digital Media.<br />
For more information on the educational uses of VisTrails see [http://www.vistrails.org/index.php/Vistrails_and_Teaching VisTrails and Teaching]. Our [http://www.cs.utah.edu/~juliana/pub/vistrails-teaching-eurographics2010.pdf paper] describing a provenance-rich teaching methodology received the Best Paper Award at the Eurographics 2010 Education program.<br />
<br />
== Sponsors ==<br />
<br />
This work has been partially supported by the National Science Foundation under grants <br />
[http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0905385 IIS-0905385],<br />
[http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0844572 IIS-0844572],<br />
[http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0746500 IIS CAREER-0746500],<br />
[http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0751152 CNS-0751152],<br />
[http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0513692 IIS-0513692], <br />
[http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0401498 CCF-0401498], <br />
[http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0541560 CNS-0541560], [http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0405402 OISE-0405402], <br />
[http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0424602 OCE-0424602], <br />
[http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0524096 CNS-0524096],<br />
[http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0534628 IIS-0534628],<br />
the Department of Energy under the SciDAC program<br />
([http://www.scidac.org/compsci/SDM.html SDM]<br />
and<br />
[http://www.scidac.org/viz/VACET.html VACET]) and UV-CDAT,<br />
IBM Faculty Awards (2005, 2006, 2007, and 2008) and a University of Utah Seed Grant.<br />
<br />
== Related Sites ==<br />
<br />
[[BirdVis]]<br />
<br />
[http://www.crowdlabs.org CrowdLabs]<br />
<br />
[[RepeatabilityCentral]]<br />
<br />
[[ProvenanceAnalytics]]<br />
<br />
[[Provenance: potpourri]]</div>Remihttps://www.vistrails.org//index.php?title=Main_Page&diff=18852Main Page2019-09-19T15:33:03Z<p>Remi: No longer maintained</p>
<hr />
<div>This is the official website for VisTrails, an open-source scientific workflow and provenance management system that supports data exploration and visualization.<br />
<br />
VisTrails provides a general infrastructure that can be combined with existing systems and libraries. Official releases come with several packages, including VTK, matplotlib, and ImageMagick. The system is written in Python/Qt and runs on Mac, Unix and Windows.<br />
<br />
[[#Downloading VisTrails|Read more >]]<br />
<br />
'''Please note that at this time, we are no longer maintaining VisTrails.'''<br />
<br />
== Downloading VisTrails ==<br />
Current and past releases are available on our [[downloads|download page]].<br />
<br />
== News ==<br />
* [2016-05-03] VisTrails 2.2.4 has been released and is available on the [[Downloads]] page.<br />
* [2015-09-11] SAHM VisTrails training course at USGS: http://pubs.usgs.gov/fs/2014/3007/pdf/fs2014-3007.pdf<br />
* [2015-07-10] VisTrails 2.2.2 has been released and is available on the [[Downloads]] page.<br />
* [2015-04-22] VisTrails 2.2 has been released and is available on the [[Downloads]] page.<br />
* [2015-03-21] VisTrails 2.1.5 has been released and is available on the [[Downloads]] page.<br />
* [2014-08-25] VisTrails 2.1.4 has been released and is available on the [[Downloads]] page.<br />
* VisTrails 2.0.3 has been released and is available on the [[Downloads]] page.<br />
* To learn more about how to explore climate data using DV3D and ParaView in UV-CDAT, check out http://www.kitware.com/source/home/post/105<br />
* Comparison of two forecasts of Hurricane Sandy using the DV3D slicer in UV-CDAT http://www.youtube.com/watch?v=BFN0RzN1hSE&feature=em-share_video_user<br />
* Presentation on the use of VisTrails at NASA: VisTrails: Workflow Management System on NEX - https://c3.nasa.gov/nex/static/media/other/Vistrails-Votava.pdf<br />
* VisTrails 2.0 has been released and it is available for download on our [[Downloads]] page. VisTrails is now distributed under the BSD license and its GUI was completely re-designed to make it easier to manage and interact with multiple workflows. It also includes a much improved [http://www.vistrails.org/usersguide Users' Manual]. See the [http://sourceforge.net/projects/vistrails/files/vistrails/v2.0/README/view release notes] for a complete list of changes. Some notable features include: <br />
** the ability to create mashups; <br />
** support for the creation of reproducible papers; <br />
** a provenance viewer; <br />
** support for visual diffs as well as queries over workflows in different vistrails; <br />
** a Control Flow Assistant to simplify the creation of workflows with loops and conditional constructs;<br />
** support for global variables, so that modules deriving from Constant can be used to create variables that are shared among multiple workflows. <br />
<br />
[[Image:Screen Shot 2012-01-12 at 2.50.19 PM.png|200px|right]]<br />
* UV-CDAT 1.0.0 has been released. More detailed information and binaries are available at http://uvcdat.llnl.gov<br />
** UV-CDAT was demoed at [http://ipaw2012.bren.ucsb.edu IPAW 2012]<br />
** Video tutorials are available at http://uv-cdat.llnl.gov/wiki/VideoTutorials<br />
** Binaries (Mac/RedHAT6/Ubuntu) or build-from-sources options can be found here: http://uv-cdat.llnl.gov/install/<br />
** Please report any issues you encounter via bugzilla at: http://uv-cdat.llnl.gov/bugzilla/<br />
<br />
<br />
* [[Older News]]<br />
<br />
== VisTrails Overview ==<br />
[[Image:Cosmology_example.png|480px|right]]<br />
VisTrails is an open-source scientific workflow and provenance management system that provides support for simulations, data exploration and visualization. Whereas workflows have been traditionally used to automate repetitive tasks, for applications that are exploratory in nature, such as simulations, data analysis and visualization, very little is repeated---change is the norm. As an engineer or scientist generates and evaluates hypotheses about data under study, a series of different, albeit related, workflows are created while a workflow is adjusted in an interactive process. VisTrails was designed to manage these rapidly-evolving workflows. <br />
<br />
A key distinguishing feature of VisTrails is a comprehensive provenance infrastructure that maintains detailed history information about the steps followed and data derived in the course of an exploratory task: VisTrails maintains provenance of data products, of the workflows that derive these products and their executions. This information is persisted as XML files or in a relational database, and it allows users to navigate workflow versions in an intuitive way, to undo changes but not lose any results, to visually compare different workflows and their results, and to examine the actions that led to a result. It also enables a series operations and user interfaces that simplify workflow design and use, including the ability to create and refine workflows by analogy and to query workflows by example.<br />
<br />
VisTrails supports the creation and execution of workflows. It allows the combination of loosely-coupled resources, specialized libraries, grid and Web services. The released version comes with support for several packages including, [http://www.vtk.org/ VTK], [http://www.imagemagick.org/script/index.php Image Magick], Web Services, and [http://www.scipy.org/PyLab pylab]. You can also download [[UserContributedPackages|packages contributed by our users]], as well as [http://www.vistrails.org/usersguide/v2.1/html/packages.html create your own packages/modules]. There are also a number of [[Projects using VisTrails|projects extending or building on top of VisTrails]].<br />
Workflows can be run interactively, through the VisTrails GUI, or in batch using a VisTrails server.<br />
<br />
VisTrails 1.0 was released in October 2007. Since then, the system has been downloaded thousands of times. VisTrails is written in Python and it uses the multi-platform Qt library for its user interface. It runs on Mac, Linux and Windows. You can download VisTrails from http://www.vistrails.org/index.php/Downloads<br />
<br />
=== Publishing Reproducible Results ===<br />
VisTrails 2.0 allows the inclusion of reproducible results in LaTeX/PDF documents. We provide a LaTeX package that allows users to add links to their results in the LaTeX source. For example: <br />
<br />
<code><br />
\usepackage{vistrails}<br />
<br />
...<br />
<br />
\begin{figure}<br />
<br />
\begin{center}<br />
<br />
\subfigure[a=0.9]{\vistrail[filename=alps.vt, version=2, pdf]{width=8cm}}<br />
<br />
\subfigure[a=0.9]{\vistrail[filename=alps.vt, version=11, pdf,buildalways]{width=8cm}}<br />
<br />
\caption{A figure produced by an ALPS VisTrails workflow. Clicking the figure retrieves the workflow used to <br />
create it. Opening that workflow on a machine with VisTrails and ALPS installed lets the reader execute the full calculation.}<br />
<br />
\end{center}<br />
<br />
\end{figure}<br />
</code><br />
<br />
Once the LaTeX document is compiled, the figure in the PDF becomes active, and when clicked, it will invoke VisTrails and reproduce the result.<br />
You can also upload your results to [http://www.crowdlabs.org CrowdLabs] and export them to Web sites or Wikis, where users can interact with them through a Web browser. See e.g., http://www.crowdlabs.org/vistrails/medleys/details/26/<br />
<br />
=== Building Workflows ===<br />
VisTrails supports workflows expressed as dataflows. It can also support functional loops and conditional statements (new in VisTrails 1.3!). In addition, VisTrails is easily extensible; users can quickly create new modules or wrap existing code or libraries for use as modules in VisTrails. Also, the <code>PythonSource</code> module can be used to write one-time translations or test code in a workflow. See the [http://sourceforge.net/projects/vistrails/files/vistrails/vistrails-usersguide-1.3-rev198.pdf/download Users' Guide] for more details on all of these features.<br />
<br />
=== Combining VisTrails Provenance with other Tools ===<br />
[[Image:DisplayWall_ipod_screenshot.png|180px|right]]<br />
The VisTrails provenance infrastructure can also be combined with interactive tools. We are developing a series of provenance plugins for different tools, including [https://wci.llnl.gov/codes/visit VisIt] and [http://www.paraview.org/ ParaView]. The [http://www.vistrails.org/index.php/ParaView_Plugin open-source ParaView plugin has been released].<br />
<br />
=== Visualizing and Comparing Workflow Results ===<br />
VisTrails provides a ''spreadsheet'' where users can compare the results of multiple workflows, or multiple workflow runs. Besides using a desktop, you can display the results of VisTrails workflows on a display wall or on an iPod. Users can also compare different workflows using the visual difference interface.<br />
<br />
=== Querying and Refining Workflows by Example ===<br />
Users can quickly construct expressive queries over a collection of workflows by example, using the same familiar interface they use to build workflows. VisTrails also provides an analogy mechanism whereby users can perform complex modifications to workflows by analogy, without having to directly modify the workflow specifications.<br />
<br />
==[[Documentation|System Documentation]]==<br />
If you want to get more information about VisTrails you can consult the Users' Manual or the FAQ (see links below). If you have questions that are not answered in these resources, you can post it to the users' mailing list: http://www.vistrails.org/index.php/MailingLists<br />
<br />
To report bugs, please use our bug tracker: https://github.com/VisTrails/VisTrails/issues<br />
<br />
=== [[Documentation#Screen Shots|Screen Shots]]===<br />
=== [[Users_Guide|User's Guide]]===<br />
A Web version of the Users Guide is currently under development and can be accessed at http://www.vistrails.org/usersguide.<br />
=== [[FAQ]]===<br />
<br />
==[[MailingLists|Mailing Lists]]==<br />
==[[Publications, Tutorials and Presentations]]==<br />
==[[People]]==<br />
<br />
== [http://www.vistrails.org/index.php/Projects_using_VisTrails Some Projects using VisTrails] ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! USGS Habitat modeling<br />
|-<br />
| [[Image:usgs.png|200px|left]]<br />
|<br />
|}<br />
<br />
{| class="wikitable"<br />
|-<br />
! NASA Climate Data Analysis <br />
! DOE CDAT<br />
! ALPS Simulations<br />
|-<br />
| [[Image:nasa.png|200px|left]]<br />
| [[Image:cdat.png|200px|left]]<br />
| [[Image:alps-shot.png|200px|left]]<br />
|<br />
|}<br />
<br />
{| class="wikitable"<br />
|-<br />
! NSF STC CMOP<br />
! NSF CDI Wildfire<br />
! NSF DataONE-EVA<br />
|-<br />
| [[Image:cmop-ss.png|200px|left]]<br />
| [[Image:wildfire.png|200px|center]]<br />
| [[Image:eva.png|200px|left]]<br />
|<br />
|}<br />
[http://www.vistrails.org/index.php/Projects_using_VisTrails See other projects]<br />
<br />
== [[Vistrails and Teaching]] ==<br />
<br />
The VisTrails system has been used a tool to teach different courses, including Scientific Visualization and Digital Media.<br />
For more information on the educational uses of VisTrails see [http://www.vistrails.org/index.php/Vistrails_and_Teaching VisTrails and Teaching]. Our [http://www.cs.utah.edu/~juliana/pub/vistrails-teaching-eurographics2010.pdf paper] describing a provenance-rich teaching methodology received the Best Paper Award at the Eurographics 2010 Education program.<br />
<br />
== Sponsors ==<br />
<br />
This work has been partially supported by the National Science Foundation under grants <br />
[http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0905385 IIS-0905385],<br />
[http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0844572 IIS-0844572],<br />
[http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0746500 IIS CAREER-0746500],<br />
[http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0751152 CNS-0751152],<br />
[http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0513692 IIS-0513692], <br />
[http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0401498 CCF-0401498], <br />
[http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0541560 CNS-0541560], [http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0405402 OISE-0405402], <br />
[http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0424602 OCE-0424602], <br />
[http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0524096 CNS-0524096],<br />
[http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0534628 IIS-0534628],<br />
the Department of Energy under the SciDAC program<br />
([http://www.scidac.org/compsci/SDM.html SDM]<br />
and<br />
[http://www.scidac.org/viz/VACET.html VACET]) and UV-CDAT,<br />
IBM Faculty Awards (2005, 2006, 2007, and 2008) and a University of Utah Seed Grant.<br />
<br />
== Related Sites ==<br />
<br />
[[BirdVis]]<br />
<br />
[http://www.crowdlabs.org CrowdLabs]<br />
<br />
[[RepeatabilityCentral]]<br />
<br />
[[ProvenanceAnalytics]]<br />
<br />
[[Provenance: potpourri]]</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=14192Development2017-06-28T15:29:19Z<p>Remi: /* June 28, 2017 */</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2017 ==<br />
<br />
=== June 28, 2017 ===<br />
<br />
'''Updates'''<br />
* [DK] Testing UUID/SQLAlchemy branch<br />
* [DK] Working on meta-versions<br />
** would be nice to have pipeline and vistrail separate<br />
** [RR] Then do we only have meta-actions, any change to the pipeline is a meta-action that adds a version that does the change?<br />
** [DK] Yes?<br />
* [RR] Notebook interface (Vizier) making good progress, figuring out cell outputs<br />
<br />
'''Items to Discuss'''<br />
<br />
=== June 7, 2017 ===<br />
<br />
'''Updates'''<br />
* [DK] Merged UUID/SQLAlchemy branch with new bundle format branch, getting rid of the bugs<br />
<br />
'''Items to Discuss'''<br />
* [RR] How to specify an "intermediate output" that should be presented to the user, from any module? (useful for notebook interface)<br />
** Can designate in module description an output port<br />
** Value has to be serializable<br />
** Should use the OutputModes work to handle multiple modes, configuration, selection, ...<br />
* [RR] Is VisTrails with SQL usable concurrently?<br />
** [DK] Probably not, demo from [TE] exists<br />
<br />
=== May 10, 2017 ===<br />
<br />
'''Items to Discuss'''<br />
<br />
* Identifiers<br />
* Bundles & DB Code<br />
* Interpreter<br />
* Merge / Versioning Version Trees<br />
* Python3<br />
* Workings of List<br />
* Packages: Wrappers + new Versions (VTK7)<br />
* Script Conversion<br />
* Intermediate Outputs: reuse output module work?<br />
** would be useful for both workflows (run to a particular checkpoint)<br />
** and useful for notebook interfaces<br />
* SQL backend: does it allow for concurrent access?<br />
** Yes, real-time collaboration work for concurrent single vistrail access<br />
** Should be able to access different vistrails at once without issues<br />
<br />
=== April 19, 2017 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
=== April 5, 2017 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
=== March 8, 2017 ===<br />
<br />
'''Updates'''<br />
* [RR] Added a looping indicator on modules [https://github.com/VisTrails/VisTrails/pull/1217 #1217]<br />
** This makes it easier to understand what's looping, no need to follow edges and count<br />
* [RR] Refactoring CLTools for integration in Docker package<br />
* [RR] Will try to refactor autogenerated package logic next (need to update TensorFlow package for TF 1.0, should really be using this)<br />
<br />
'''Items to Discuss'''<br />
* [RR] Problem with vtkRendererToSpreadsheet output mode: does it's own ModuleConnector handling, interfering with looping/streaming logic, to add the module ID on the renderers [https://github.com/VisTrails/VisTrails/issues/1216 #1216]<br />
** Can we move this up to the renderer modules?<br />
** Thankfully RR couldn't find much of this madness in the rest of code (outside of controlflow and parallelflow)<br />
<br />
=== February 24, 2017 ===<br />
<br />
'''Updates'''<br />
* [RR] Rich-text module documentation ([https://github.com/VisTrails/VisTrails/pull/1210 #1210])<br />
** Went over packages and fixed format; maybe we should have a test to validate RST?<br />
* [RR] Single-instance shutdown with <tt>--remote-shutdown</tt> ([https://github.com/VisTrails/VisTrails/pull/1204 #1204])<br />
* [RR] Bundle installation not super reliable; probably need to go in and add checks for every usual environment<br />
<br />
'''Items to Discuss'''<br />
* [RR] There are issues with list-depth<br />
** Some VTK ports marked with depth=1 don't actually accept multiple inputs (vtkDataSetMapper)<br />
** Issue with looping groups<br />
<br />
=== January 25, 2017 ===<br />
<br />
'''Updates'''<br />
<br />
<br />
'''Items to Discuss'''<br />
<br />
* Scaling VisTrails<br />
* sql-alchemy, uuid, bundle branches/merges<br />
<br />
=== January 11, 2017 ===<br />
<br />
'''Updates'''<br />
* [RR] New VisTrails version in Debian [https://packages.qa.debian.org/v/vistrails.html]<br />
<br />
'''Items to Discuss'''<br />
* [RR] Collaboration with school of medicine to port HiC-bench to VisTrails<br />
** Have homemade workflow system based on R scripts and symlinks<br />
*** Each steps has multiple sets of parameters<br />
*** Submits PBS jobs for each step with each set of parameters, using PBS job metadata to wait on previous step<br />
*** Gets all results in directory structure <code>step3-parameters/step2-parameters/step1-parameters/files</code><br />
** VisTrails improves provenance capture and makes editing pipelines easier<br />
*** Need better support for aggregation (groupby operation might be good enough; pandas?)<br />
<br />
== 2016 ==<br />
=== December 14, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Single-instance code: over-engineered, but will work<br />
** (Have to avoid race conditions on startup, and OSes offer very different locking primitives)<br />
<br />
'''Items to Discuss'''<br />
* Email from Matt Dirks: combobox should work, ask for more code?<br />
* Email from marin.nl:<br />
** looks like they would benefit from new interpreter<br />
** how to store their consolidated provenance when they exchange .vt files around?<br />
<br />
=== November 30, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Working on single instance code for MARIN ([https://github.com/VisTrails/VisTrails/pull/1205 #1205])<br />
** Remote shutdown ready ([https://github.com/VisTrails/VisTrails/pull/1204 #1204])<br />
** Still improving it<br />
** Want to return something more structured to the client (success/fail code, stdout, stderr)<br />
** Should be usable without Qt<br />
* [DK] Shutting down VisTrails has issues: cleans up interpreter (to remove files) but also resets persistent pipeline for no reason<br />
* [DK] Working on batch execution using console_mode<br />
* [DK] Settings aliased parameters when using core.api?<br />
<br />
=== November 16, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [RR] Issue [https://github.com/VisTrails/VisTrails/issues/1200 #1200] from @samtux: need to set PyQt to API version 2 before ever importing it; using require_pyqt_api2() does that (also can install bundle)<br />
<br />
'''Items to Discuss'''<br />
<br />
* Email from Pieter: executing pipelines with parameters from command-line using single-instance<br />
** Seems to work, main instance gets full command-line and alters configuration temporarily<br />
** Not sure what his error is (VisTrails returns little information in a lot of cases); though execution error should return something more meaningful<br />
** Actions:<br />
*** Can add switch to kill main instance<br />
*** Can improve single-instance code to return more meaningful errors + machine-readable output from "client" VisTrails<br />
*** Add client/server code to <tt>core</tt> that doesn't use Qt? Also need a run.py-line entrypoint that doesn't use gui (calls console mode like <tt>gui.application.VistrailsApplicationSingleton#noninteractiveMode()</tt>)?<br />
<br />
=== November 2, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* New version 2.2-2 of Debian package, still [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=796994 wrong copyright], and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=748692 broken bundle installation]<br />
** Finally replied, says copyright issue will be fixed soon, working on including VisTrails 2.2.4 for Debian Stretch<br />
* Working on Docker package, low hanging fruit because we have CLTools, and useful for some workflow (also step towards CWL support)<br />
<br />
'''Items to Discuss'''<br />
<br />
=== October 5, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
<br />
* working toward a 3.0 release; [https://github.com/VisTrails/VisTrails/projects/1 project board]<br />
** unique ids<br />
** bundles<br />
** GUI fixes<br />
** wrapping updates<br />
** python3<br />
** interpreter?<br />
<br />
* command-line input parameters to provenance<br />
** [DK] we don't create a persisted workflow from those input params, i think<br />
** [DK] these are already defined as aliases so we treat them differently form a normal input param<br />
** [DK] still have the potential problem of allowing a user to view the workflow as it was run with a given set of aliases<br />
** Do the vistrails variables values get stored in provenance? (what if values change?)<br />
<br />
=== September 14, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] [http://aosabook.org/en/vistrails.html VisTrails] in [http://aosabook.org/ The Architecture of Open Source Applications] Vol I (2012)<br />
** Did we know/contribute to this?<br />
*** [DK] Yes. We wrote the text for the VisTrails chapter<br />
** Drafted response<br />
*** [DK] Looks good, will forward.<br />
<br />
=== August 31, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Went over vgc.poly.edu/vistrails.org config:<br />
** Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)<br />
** Fixed usersguide scripts (won't disappear again)<br />
** Moved some websites over to HTTPS<br />
** [https://docs.google.com/document/d/18YZewjR2YY9cQ9pzdA-nU9jbu2L7jYcPlDi3iaPadwA/edit Websites]<br />
* [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs ([http://vistrails.readthedocs.io/ currently there]) and users' guide under the same RTD "project"<br />
* [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that<br />
<br />
'''Items to Discuss'''<br />
* [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/<br />
** Get rid of more services? (svn, trac, ...)<br />
* [RR] Notebook execution works. How do we support more languages (than Python)?<br />
** Need: detection, metadata reader, code to input values<br />
* [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?<br />
<br />
=== August 17, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Fixed [https://github.com/VisTrails/VisTrails/commit/2cd42d156243067a7663bb0bf0156e5d87188c2f parameter order upgrade bug] that breaks VTK examples.<br />
* [TE] Matplotlib and sklearn are exportable using module-to-script api<br />
* [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.<br />
<br />
'''Items to Discuss'''<br />
* Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!<br />
* Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)<br />
* [TE] Wrap-up notes<br />
<br />
=== August 10, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Python Wrapper and scripting<br />
** All changes merged into python-wrapper branch<br />
*** Tests pass<br />
*** Still need to update sub-branches<br />
** Minor fixes<br />
*** vtkPlot3DReader upgrade bug<br />
*** Added tuple exporting<br />
** Most of vtk examples can be exported [https://gist.github.com/rexissimus/5ac7374cde8525d84f9443f019d234e2 example]<br />
*** Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary<br />
**** Could be a bug in the old vistrail, but not sure<br />
* [DK] Subworkflows and uuids<br />
<br />
=== August 3, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Wrapper patches now stored as [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/vtk_patch.py python code]<br />
** Connection variables now lowercase<br />
** Can we use code as output variables?<br />
*** E.g. "vtk_dataset_mapper.GetOutput(0)"<br />
*** Could mutate output when output method is only supposed to be called once<br />
*** Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)<br />
<br />
=== July 27, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] DB bundle serialization implemented<br />
<br />
'''Items to Discuss'''<br />
* [TE] Finished using patch in script [https://github.com/VisTrails/VisTrails/tree/python-wrapper-script-patches python-wrapper-script-patches]<br />
** locale?<br />
** [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d new patching example] vs [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 old patching example]<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/parse.py#L135-L273 Example of new patches] <br />
** Legacy '.vtkInstance' is now added as a patch "self.vtkInstance=self" (see new patching example)<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/api.py New api for executing standalone modules]<br />
*** Accessing module was disabled for core/api, but a different api for this is probably better anyway.<br />
*** api.Package now gives [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d#file-new_patching-py-L26 directly callable module compute functions]<br />
*** Only works on modules not using interpreter functionality ('.interpreter', etc.)<br />
**** for zip file modules and others that need temp files and use the interpreter for this, need to modify the module, but could work around with python libraries for managing temp files<br />
** Fixed more renaming issues - [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/scripts.py#L63-L64 function argument names should not be replaced]<br />
** Still need to do something nicer for the patch formatting<br />
*** patch keys currently contain non-python-variable characters but this can be fixed<br />
*** Need to replace string template with python variables that is replacable using RedBaron<br />
**** Patches could then look like:<br />
<br />
def vtkInstance():<br />
self.vtkInstance = self<br />
<br />
=== July 20, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] New patch method for python-wrapper [https://gist.github.com/rexissimus/009b303e4cadcd4b8f2fec6cf848cb2b example]<br />
** A patch is a string template with self/input/output variables for patching a class method <br />
** Patches are stored in xml specification<br />
** Patches are assigned to methods in ClassSpec's<br />
** Used both for execution in vistrails and when exporting as script<br />
*** Executed with "exec" when run in vistrails<br />
*** Turns patches into natural calls in script<br />
** Wrapping VTK library no longer needed<br />
*** Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)<br />
** Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)<br />
** Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)<br />
** TODO's<br />
*** Add to export_as_script<br />
*** Patch Modules that represent Python functions as well<br />
<br />
* [vistrails-users] Read data from SQL example?<br />
<br />
=== July 13, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] bundles on sql-alchemy branch<br />
** Bundle, BundleMapping, and BundleSerializer<br />
** New BaseSerializers to deal with versions of directory, zip, db serializers<br />
** Can register the BundleSerializers with the base serializers<br />
** Translation between bundle versions<br />
** Bundle.add_object "just works" if an appropriate mapping was registered<br />
<br />
'''Items to Discuss'''<br />
* [TE] Simplified running module using API for scripting [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 example]<br />
** API Module can be executed using '''compute''' method<br />
*** Takes list of outputs to compute + inputs as keyword arguments<br />
* Status of general cell synchronization (vistrails-users)<br />
<br />
=== July 6, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] Working on bundles<br />
<br />
'''Items to Discuss'''<br />
* [TE] Using vistrails API in scripting [https://gist.github.com/rexissimus/034d9c4dedc3299a079e84981dedd8c7 example 1 (API with VTK)] [https://gist.github.com/rexissimus/a4a9ca6c08d4058c64967d2b35415fd5 example 2 (DownloadFile only)]<br />
** Call module in script using vistrails API<br />
** Can be used when no export method is defined.<br />
* [TE] [https://github.com/VisTrails/VisTrails/commit/c76c109349769fee1280f85c22678380a2f9cd39 Server news bug] Critical?<br />
**[RR] No, unreleased<br />
* [DK] New bundle organization<br />
** BundleMapping contains individual mappings from raw objects to BundleObj<br />
** Must be separate in order to have in-memory, separate from serializer settings<br />
<br />
=== June 29, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as python script<br />
** Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) [https://gist.github.com/rexissimus/16caf5200802e150296f62a0c7844c5a example]<br />
** Preludes (imports) still need improvements<br />
** Works with vtkInteractionHandler (clipping plane)<br />
** Resulting clipping plane is incorrect? (supposed to hide one side of the plane)<br />
** vtkRendererOutput Uses VTKRenderWindow as output<br />
*** Should support different output modes?<br />
*** Does not work with multiple outputs (Next cell is displayed when current is closed)<br />
*** Uses wrapped vtkInstance (as vvtk)<br />
**** Create new patch system with code that can be used both in vistrails and in a python script?<br />
<br />
=== June 22, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fix for upgraded analogies (PR [https://github.com/VisTrails/VisTrails/pull/1177 #1177])<br />
** Uses module location to find module remap (The best we can do right now?)<br />
** Also fixes remapped port names for connections to existing modules <br />
** May still fail on complex upgrades like adding a function on an upgraded module that has a different port name<br />
<br />
* [TE] Working on exporting wrapped libraries (VTK)<br />
<br />
=== June 15, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Added depth logic to python script exporter ([https://github.com/VisTrails/VisTrails/commit/50f220e4f6abe5a7bb8689f73f456f43df2dbc12 full commit])<br />
** Fully implements connection merging and list looping<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Making analogy work on upgraded workflows ([https://github.com/VisTrails/VisTrails/issues/1175 #1175])<br />
** Analogy only works We need to use the non-upgraded action chain<br />
** But pipelines need to be valid (matcher uses module portspecs)<br />
** Tried to get upgrade remap from upgrade action, but this seems impossible<br />
** Use module location instead?<br />
*** Location should be identical for one-to-one module upgrades<br />
<br />
=== June 1, 2016 ===<br />
<br />
'''Updates'''<br />
* VisTrails 2.2.4 announced (finally)<br />
** 5 survey replies<br />
** 47 downloads last week<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as script (python-wrapper)<br />
** to_python_script<br />
*** needs the pipeline (or at least the connections)<br />
**** Should not generate code for any possible VTK method (exporter detects which ports is used in the source)<br />
**** VTK needs to know connection order<br />
**** Adding functions directly to the script results in nicer code<br />
** Add support for multiple input connections<br />
** Add support for list looping<br />
<br />
=== May 25, 2016 ===<br />
<br />
'''Updates'''<br />
* Mailing-list fixed<br />
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])<br />
<br />
'''Items to Discuss'''<br />
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)<br />
* [TE] Union ports<br />
** Show union ports in pipeline (DEMO)<br />
<br />
=== May 18, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] mailing list still down - Julio working on it<br />
* [TE] Improved union port<br />
** New schema 1.0.5 adds portitem.union string attribute<br />
** Works with pipeline view, parameter exploration view, and mashup view<br />
** Should work well for matplotlib<br />
** Use with VTK ports?<br />
*** It uses [item_1,item_2] style ports usually with different tuple length.<br />
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]<br />
<br />
=== May 11, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?<br />
*** Shows single port for all ports with same sort_key<br />
*** Select specific type when adding/showing<br />
*** Only one type should be visible on the module at any time<br />
*** Making tuple parameters union types is probably too complicated<br />
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly) <br />
<br />
* [TE] Feature suggestion: Let connections connect to functions/parameters<br />
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters<br />
** Use separate views for ports and functions (like before)<br />
*** Allow rearranging of functions<br />
** Pros:<br />
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.<br />
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.<br />
**** Makes List module simpler<br />
*** Connect to parameter in tuple<br />
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds. <br />
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.<br />
** Cons:<br />
*** Interface more complicated and crowded<br />
*** More difficult for users to grasp?<br />
*** No immediate use case<br />
*** Makes no union type for tuple parameters a bigger problem.<br />
<br />
* [TE] Publish user survey?<br />
** No response from vistrails release yet<br />
** Not announced on vistrails-users yet.<br />
<br />
=== May 4, 2016 ===<br />
'''Updates'''<br />
* VisTrails 2.2.4 released<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
<br />
=== April 27, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Wrapping Bokeh<br />
** New method type "operation"<br />
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output<br />
**** Glyph sets function name and arguments as output<br />
***** Not serializable, but should always be computed together with the subject of the operation.<br />
**** Figure applies Glyph operation on itself<br />
*** Useful for matplotlib?<br />
**** matplotlib currently find actors by searching upstream?<br />
**** But this is more difficult since matplotlib is not function-based.<br />
** New "union" port type<br />
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value<br />
**** Uses only first value in tuple<br />
**** Bokeh have many properties that support (list, string, Float)<br />
*** Does not work with depth=1 types (e.g. Float of depth 1)<br />
**** Turn Depth 1 types with known type into lists?<br />
<br />
* [TE] New bugfix release<br />
** Finalize survey<br />
<br />
=== April 20, 2016 ===<br />
'''Updates'''<br />
* Merged upgrade fixes<br />
** Now has infinite loop test<br />
** loop test now correctly loops 5 times<br />
*** remap_module used wrong (current) package version when creating ModuleDescriptor<br />
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])<br />
<br />
'''Items to Discuss'''<br />
* New bugfix release<br />
** Merge usagestats<br />
** Need to finalize survey<br />
<br />
=== April 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?<br />
<br />
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])<br />
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]<br />
** Cherry-pick test to preserve source?<br />
** Need maxPipelineFixAttempts option?<br />
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.<br />
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?<br />
*** Would need to be moved into handle_invalid_pipeline<br />
<br />
=== April 6, 2016 ===<br />
'''Updates'''<br />
* [TE] Query View improvements<br />
** Done except for displaying "Version Info" and #1164<br />
* [RR] Working on finishing new interpreter (finally!)<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Try latest upgrade first, then previous.<br />
** If all fail, select latest upgrade?<br />
<br />
=== March 30, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Query View improvements<br />
** Unified do_version_switch and get_upgrades as validate_version()<br />
** Problems with query view using its own controller<br />
*** Fixed module info not being displayed for query results<br />
*** Still no version info for query results<br />
*** Still an issue with version query view not being updated<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== Older meetings ==<br />
* [[Development/2015]]<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=14075Development2017-06-07T15:30:21Z<p>Remi: /* June 7, 2017 */</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2017 ==<br />
<br />
=== June 7, 2017 ===<br />
<br />
'''Updates'''<br />
* [DK] Merged UUID/SQLAlchemy branch with new bundle format branch, getting rid of the bugs<br />
<br />
'''Items to Discuss'''<br />
* [RR] How to specify an "intermediate output" that should be presented to the user, from any module? (useful for notebook interface)<br />
** Can designate in module description an output port<br />
** Value has to be serializable<br />
** Should use the OutputModes work to handle multiple modes, configuration, selection, ...<br />
* [RR] Is VisTrails with SQL usable concurrently?<br />
** [DK] Probably not, demo from [TE] exists<br />
<br />
=== May 10, 2017 ===<br />
<br />
'''Items to Discuss'''<br />
<br />
* Identifiers<br />
* Bundles & DB Code<br />
* Interpreter<br />
* Merge / Versioning Version Trees<br />
* Python3<br />
* Workings of List<br />
* Packages: Wrappers + new Versions (VTK7)<br />
* Script Conversion<br />
* Intermediate Outputs: reuse output module work?<br />
** would be useful for both workflows (run to a particular checkpoint)<br />
** and useful for notebook interfaces<br />
* SQL backend: does it allow for concurrent access?<br />
** Yes, real-time collaboration work for concurrent single vistrail access<br />
** Should be able to access different vistrails at once without issues<br />
<br />
=== April 19, 2017 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
=== April 5, 2017 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
=== March 8, 2017 ===<br />
<br />
'''Updates'''<br />
* [RR] Added a looping indicator on modules [https://github.com/VisTrails/VisTrails/pull/1217 #1217]<br />
** This makes it easier to understand what's looping, no need to follow edges and count<br />
* [RR] Refactoring CLTools for integration in Docker package<br />
* [RR] Will try to refactor autogenerated package logic next (need to update TensorFlow package for TF 1.0, should really be using this)<br />
<br />
'''Items to Discuss'''<br />
* [RR] Problem with vtkRendererToSpreadsheet output mode: does it's own ModuleConnector handling, interfering with looping/streaming logic, to add the module ID on the renderers [https://github.com/VisTrails/VisTrails/issues/1216 #1216]<br />
** Can we move this up to the renderer modules?<br />
** Thankfully RR couldn't find much of this madness in the rest of code (outside of controlflow and parallelflow)<br />
<br />
=== February 24, 2017 ===<br />
<br />
'''Updates'''<br />
* [RR] Rich-text module documentation ([https://github.com/VisTrails/VisTrails/pull/1210 #1210])<br />
** Went over packages and fixed format; maybe we should have a test to validate RST?<br />
* [RR] Single-instance shutdown with <tt>--remote-shutdown</tt> ([https://github.com/VisTrails/VisTrails/pull/1204 #1204])<br />
* [RR] Bundle installation not super reliable; probably need to go in and add checks for every usual environment<br />
<br />
'''Items to Discuss'''<br />
* [RR] There are issues with list-depth<br />
** Some VTK ports marked with depth=1 don't actually accept multiple inputs (vtkDataSetMapper)<br />
** Issue with looping groups<br />
<br />
=== January 25, 2017 ===<br />
<br />
'''Updates'''<br />
<br />
<br />
'''Items to Discuss'''<br />
<br />
* Scaling VisTrails<br />
* sql-alchemy, uuid, bundle branches/merges<br />
<br />
=== January 11, 2017 ===<br />
<br />
'''Updates'''<br />
* [RR] New VisTrails version in Debian [https://packages.qa.debian.org/v/vistrails.html]<br />
<br />
'''Items to Discuss'''<br />
* [RR] Collaboration with school of medicine to port HiC-bench to VisTrails<br />
** Have homemade workflow system based on R scripts and symlinks<br />
*** Each steps has multiple sets of parameters<br />
*** Submits PBS jobs for each step with each set of parameters, using PBS job metadata to wait on previous step<br />
*** Gets all results in directory structure <code>step3-parameters/step2-parameters/step1-parameters/files</code><br />
** VisTrails improves provenance capture and makes editing pipelines easier<br />
*** Need better support for aggregation (groupby operation might be good enough; pandas?)<br />
<br />
== 2016 ==<br />
=== December 14, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Single-instance code: over-engineered, but will work<br />
** (Have to avoid race conditions on startup, and OSes offer very different locking primitives)<br />
<br />
'''Items to Discuss'''<br />
* Email from Matt Dirks: combobox should work, ask for more code?<br />
* Email from marin.nl:<br />
** looks like they would benefit from new interpreter<br />
** how to store their consolidated provenance when they exchange .vt files around?<br />
<br />
=== November 30, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Working on single instance code for MARIN ([https://github.com/VisTrails/VisTrails/pull/1205 #1205])<br />
** Remote shutdown ready ([https://github.com/VisTrails/VisTrails/pull/1204 #1204])<br />
** Still improving it<br />
** Want to return something more structured to the client (success/fail code, stdout, stderr)<br />
** Should be usable without Qt<br />
* [DK] Shutting down VisTrails has issues: cleans up interpreter (to remove files) but also resets persistent pipeline for no reason<br />
* [DK] Working on batch execution using console_mode<br />
* [DK] Settings aliased parameters when using core.api?<br />
<br />
=== November 16, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [RR] Issue [https://github.com/VisTrails/VisTrails/issues/1200 #1200] from @samtux: need to set PyQt to API version 2 before ever importing it; using require_pyqt_api2() does that (also can install bundle)<br />
<br />
'''Items to Discuss'''<br />
<br />
* Email from Pieter: executing pipelines with parameters from command-line using single-instance<br />
** Seems to work, main instance gets full command-line and alters configuration temporarily<br />
** Not sure what his error is (VisTrails returns little information in a lot of cases); though execution error should return something more meaningful<br />
** Actions:<br />
*** Can add switch to kill main instance<br />
*** Can improve single-instance code to return more meaningful errors + machine-readable output from "client" VisTrails<br />
*** Add client/server code to <tt>core</tt> that doesn't use Qt? Also need a run.py-line entrypoint that doesn't use gui (calls console mode like <tt>gui.application.VistrailsApplicationSingleton#noninteractiveMode()</tt>)?<br />
<br />
=== November 2, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* New version 2.2-2 of Debian package, still [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=796994 wrong copyright], and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=748692 broken bundle installation]<br />
** Finally replied, says copyright issue will be fixed soon, working on including VisTrails 2.2.4 for Debian Stretch<br />
* Working on Docker package, low hanging fruit because we have CLTools, and useful for some workflow (also step towards CWL support)<br />
<br />
'''Items to Discuss'''<br />
<br />
=== October 5, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
<br />
* working toward a 3.0 release; [https://github.com/VisTrails/VisTrails/projects/1 project board]<br />
** unique ids<br />
** bundles<br />
** GUI fixes<br />
** wrapping updates<br />
** python3<br />
** interpreter?<br />
<br />
* command-line input parameters to provenance<br />
** [DK] we don't create a persisted workflow from those input params, i think<br />
** [DK] these are already defined as aliases so we treat them differently form a normal input param<br />
** [DK] still have the potential problem of allowing a user to view the workflow as it was run with a given set of aliases<br />
** Do the vistrails variables values get stored in provenance? (what if values change?)<br />
<br />
=== September 14, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] [http://aosabook.org/en/vistrails.html VisTrails] in [http://aosabook.org/ The Architecture of Open Source Applications] Vol I (2012)<br />
** Did we know/contribute to this?<br />
*** [DK] Yes. We wrote the text for the VisTrails chapter<br />
** Drafted response<br />
*** [DK] Looks good, will forward.<br />
<br />
=== August 31, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Went over vgc.poly.edu/vistrails.org config:<br />
** Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)<br />
** Fixed usersguide scripts (won't disappear again)<br />
** Moved some websites over to HTTPS<br />
** [https://docs.google.com/document/d/18YZewjR2YY9cQ9pzdA-nU9jbu2L7jYcPlDi3iaPadwA/edit Websites]<br />
* [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs ([http://vistrails.readthedocs.io/ currently there]) and users' guide under the same RTD "project"<br />
* [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that<br />
<br />
'''Items to Discuss'''<br />
* [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/<br />
** Get rid of more services? (svn, trac, ...)<br />
* [RR] Notebook execution works. How do we support more languages (than Python)?<br />
** Need: detection, metadata reader, code to input values<br />
* [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?<br />
<br />
=== August 17, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Fixed [https://github.com/VisTrails/VisTrails/commit/2cd42d156243067a7663bb0bf0156e5d87188c2f parameter order upgrade bug] that breaks VTK examples.<br />
* [TE] Matplotlib and sklearn are exportable using module-to-script api<br />
* [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.<br />
<br />
'''Items to Discuss'''<br />
* Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!<br />
* Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)<br />
* [TE] Wrap-up notes<br />
<br />
=== August 10, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Python Wrapper and scripting<br />
** All changes merged into python-wrapper branch<br />
*** Tests pass<br />
*** Still need to update sub-branches<br />
** Minor fixes<br />
*** vtkPlot3DReader upgrade bug<br />
*** Added tuple exporting<br />
** Most of vtk examples can be exported [https://gist.github.com/rexissimus/5ac7374cde8525d84f9443f019d234e2 example]<br />
*** Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary<br />
**** Could be a bug in the old vistrail, but not sure<br />
* [DK] Subworkflows and uuids<br />
<br />
=== August 3, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Wrapper patches now stored as [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/vtk_patch.py python code]<br />
** Connection variables now lowercase<br />
** Can we use code as output variables?<br />
*** E.g. "vtk_dataset_mapper.GetOutput(0)"<br />
*** Could mutate output when output method is only supposed to be called once<br />
*** Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)<br />
<br />
=== July 27, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] DB bundle serialization implemented<br />
<br />
'''Items to Discuss'''<br />
* [TE] Finished using patch in script [https://github.com/VisTrails/VisTrails/tree/python-wrapper-script-patches python-wrapper-script-patches]<br />
** locale?<br />
** [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d new patching example] vs [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 old patching example]<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/parse.py#L135-L273 Example of new patches] <br />
** Legacy '.vtkInstance' is now added as a patch "self.vtkInstance=self" (see new patching example)<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/api.py New api for executing standalone modules]<br />
*** Accessing module was disabled for core/api, but a different api for this is probably better anyway.<br />
*** api.Package now gives [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d#file-new_patching-py-L26 directly callable module compute functions]<br />
*** Only works on modules not using interpreter functionality ('.interpreter', etc.)<br />
**** for zip file modules and others that need temp files and use the interpreter for this, need to modify the module, but could work around with python libraries for managing temp files<br />
** Fixed more renaming issues - [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/scripts.py#L63-L64 function argument names should not be replaced]<br />
** Still need to do something nicer for the patch formatting<br />
*** patch keys currently contain non-python-variable characters but this can be fixed<br />
*** Need to replace string template with python variables that is replacable using RedBaron<br />
**** Patches could then look like:<br />
<br />
def vtkInstance():<br />
self.vtkInstance = self<br />
<br />
=== July 20, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] New patch method for python-wrapper [https://gist.github.com/rexissimus/009b303e4cadcd4b8f2fec6cf848cb2b example]<br />
** A patch is a string template with self/input/output variables for patching a class method <br />
** Patches are stored in xml specification<br />
** Patches are assigned to methods in ClassSpec's<br />
** Used both for execution in vistrails and when exporting as script<br />
*** Executed with "exec" when run in vistrails<br />
*** Turns patches into natural calls in script<br />
** Wrapping VTK library no longer needed<br />
*** Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)<br />
** Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)<br />
** Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)<br />
** TODO's<br />
*** Add to export_as_script<br />
*** Patch Modules that represent Python functions as well<br />
<br />
* [vistrails-users] Read data from SQL example?<br />
<br />
=== July 13, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] bundles on sql-alchemy branch<br />
** Bundle, BundleMapping, and BundleSerializer<br />
** New BaseSerializers to deal with versions of directory, zip, db serializers<br />
** Can register the BundleSerializers with the base serializers<br />
** Translation between bundle versions<br />
** Bundle.add_object "just works" if an appropriate mapping was registered<br />
<br />
'''Items to Discuss'''<br />
* [TE] Simplified running module using API for scripting [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 example]<br />
** API Module can be executed using '''compute''' method<br />
*** Takes list of outputs to compute + inputs as keyword arguments<br />
* Status of general cell synchronization (vistrails-users)<br />
<br />
=== July 6, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] Working on bundles<br />
<br />
'''Items to Discuss'''<br />
* [TE] Using vistrails API in scripting [https://gist.github.com/rexissimus/034d9c4dedc3299a079e84981dedd8c7 example 1 (API with VTK)] [https://gist.github.com/rexissimus/a4a9ca6c08d4058c64967d2b35415fd5 example 2 (DownloadFile only)]<br />
** Call module in script using vistrails API<br />
** Can be used when no export method is defined.<br />
* [TE] [https://github.com/VisTrails/VisTrails/commit/c76c109349769fee1280f85c22678380a2f9cd39 Server news bug] Critical?<br />
**[RR] No, unreleased<br />
* [DK] New bundle organization<br />
** BundleMapping contains individual mappings from raw objects to BundleObj<br />
** Must be separate in order to have in-memory, separate from serializer settings<br />
<br />
=== June 29, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as python script<br />
** Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) [https://gist.github.com/rexissimus/16caf5200802e150296f62a0c7844c5a example]<br />
** Preludes (imports) still need improvements<br />
** Works with vtkInteractionHandler (clipping plane)<br />
** Resulting clipping plane is incorrect? (supposed to hide one side of the plane)<br />
** vtkRendererOutput Uses VTKRenderWindow as output<br />
*** Should support different output modes?<br />
*** Does not work with multiple outputs (Next cell is displayed when current is closed)<br />
*** Uses wrapped vtkInstance (as vvtk)<br />
**** Create new patch system with code that can be used both in vistrails and in a python script?<br />
<br />
=== June 22, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fix for upgraded analogies (PR [https://github.com/VisTrails/VisTrails/pull/1177 #1177])<br />
** Uses module location to find module remap (The best we can do right now?)<br />
** Also fixes remapped port names for connections to existing modules <br />
** May still fail on complex upgrades like adding a function on an upgraded module that has a different port name<br />
<br />
* [TE] Working on exporting wrapped libraries (VTK)<br />
<br />
=== June 15, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Added depth logic to python script exporter ([https://github.com/VisTrails/VisTrails/commit/50f220e4f6abe5a7bb8689f73f456f43df2dbc12 full commit])<br />
** Fully implements connection merging and list looping<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Making analogy work on upgraded workflows ([https://github.com/VisTrails/VisTrails/issues/1175 #1175])<br />
** Analogy only works We need to use the non-upgraded action chain<br />
** But pipelines need to be valid (matcher uses module portspecs)<br />
** Tried to get upgrade remap from upgrade action, but this seems impossible<br />
** Use module location instead?<br />
*** Location should be identical for one-to-one module upgrades<br />
<br />
=== June 1, 2016 ===<br />
<br />
'''Updates'''<br />
* VisTrails 2.2.4 announced (finally)<br />
** 5 survey replies<br />
** 47 downloads last week<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as script (python-wrapper)<br />
** to_python_script<br />
*** needs the pipeline (or at least the connections)<br />
**** Should not generate code for any possible VTK method (exporter detects which ports is used in the source)<br />
**** VTK needs to know connection order<br />
**** Adding functions directly to the script results in nicer code<br />
** Add support for multiple input connections<br />
** Add support for list looping<br />
<br />
=== May 25, 2016 ===<br />
<br />
'''Updates'''<br />
* Mailing-list fixed<br />
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])<br />
<br />
'''Items to Discuss'''<br />
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)<br />
* [TE] Union ports<br />
** Show union ports in pipeline (DEMO)<br />
<br />
=== May 18, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] mailing list still down - Julio working on it<br />
* [TE] Improved union port<br />
** New schema 1.0.5 adds portitem.union string attribute<br />
** Works with pipeline view, parameter exploration view, and mashup view<br />
** Should work well for matplotlib<br />
** Use with VTK ports?<br />
*** It uses [item_1,item_2] style ports usually with different tuple length.<br />
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]<br />
<br />
=== May 11, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?<br />
*** Shows single port for all ports with same sort_key<br />
*** Select specific type when adding/showing<br />
*** Only one type should be visible on the module at any time<br />
*** Making tuple parameters union types is probably too complicated<br />
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly) <br />
<br />
* [TE] Feature suggestion: Let connections connect to functions/parameters<br />
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters<br />
** Use separate views for ports and functions (like before)<br />
*** Allow rearranging of functions<br />
** Pros:<br />
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.<br />
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.<br />
**** Makes List module simpler<br />
*** Connect to parameter in tuple<br />
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds. <br />
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.<br />
** Cons:<br />
*** Interface more complicated and crowded<br />
*** More difficult for users to grasp?<br />
*** No immediate use case<br />
*** Makes no union type for tuple parameters a bigger problem.<br />
<br />
* [TE] Publish user survey?<br />
** No response from vistrails release yet<br />
** Not announced on vistrails-users yet.<br />
<br />
=== May 4, 2016 ===<br />
'''Updates'''<br />
* VisTrails 2.2.4 released<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
<br />
=== April 27, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Wrapping Bokeh<br />
** New method type "operation"<br />
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output<br />
**** Glyph sets function name and arguments as output<br />
***** Not serializable, but should always be computed together with the subject of the operation.<br />
**** Figure applies Glyph operation on itself<br />
*** Useful for matplotlib?<br />
**** matplotlib currently find actors by searching upstream?<br />
**** But this is more difficult since matplotlib is not function-based.<br />
** New "union" port type<br />
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value<br />
**** Uses only first value in tuple<br />
**** Bokeh have many properties that support (list, string, Float)<br />
*** Does not work with depth=1 types (e.g. Float of depth 1)<br />
**** Turn Depth 1 types with known type into lists?<br />
<br />
* [TE] New bugfix release<br />
** Finalize survey<br />
<br />
=== April 20, 2016 ===<br />
'''Updates'''<br />
* Merged upgrade fixes<br />
** Now has infinite loop test<br />
** loop test now correctly loops 5 times<br />
*** remap_module used wrong (current) package version when creating ModuleDescriptor<br />
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])<br />
<br />
'''Items to Discuss'''<br />
* New bugfix release<br />
** Merge usagestats<br />
** Need to finalize survey<br />
<br />
=== April 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?<br />
<br />
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])<br />
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]<br />
** Cherry-pick test to preserve source?<br />
** Need maxPipelineFixAttempts option?<br />
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.<br />
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?<br />
*** Would need to be moved into handle_invalid_pipeline<br />
<br />
=== April 6, 2016 ===<br />
'''Updates'''<br />
* [TE] Query View improvements<br />
** Done except for displaying "Version Info" and #1164<br />
* [RR] Working on finishing new interpreter (finally!)<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Try latest upgrade first, then previous.<br />
** If all fail, select latest upgrade?<br />
<br />
=== March 30, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Query View improvements<br />
** Unified do_version_switch and get_upgrades as validate_version()<br />
** Problems with query view using its own controller<br />
*** Fixed module info not being displayed for query results<br />
*** Still no version info for query results<br />
*** Still an issue with version query view not being updated<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== Older meetings ==<br />
* [[Development/2015]]<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=FAQ&diff=13971FAQ2017-05-26T13:59:01Z<p>Remi: /* How do I open and execute workflows in a standard python shell? */</p>
<hr />
<div>Also check our [[known_issues|Known Issues]] page for troubleshooting.<br />
<br />
----<br />
== Running workflows ==<br />
<br />
=== How can I run a workflow using the command line? ===<br />
<br />
('''Updated for version 1.2''') Call vistrails using the following options:<br />
<br />
<code>python vistrails.py -b path_to_vistrails_file:pipeline </code><br />
<br />
where pipeline can be a version '''tag name''' or version '''id'''<br />
<br />
'''NOTE:''' If you downloaded the MacOS X bundle, you can run vistrails from the command line via the following commands in the Terminal. Change the current directory to wherever VisTrails was installed (often <code>/Applications</code>), and then type:<br />
<br />
<code>Vistrails.app/Contents/MacOS/vistrails [<cmd_line_options>]</code> [http://www.vistrails.org/usersguide/v2.1/html/batch.html#running-a-specific-workflow-in-batch-mode Running a Specific Workflow in Batch Mode]<br />
<br />
=== Using the command line, we'd like to execute a workflow multiple times, with slightly different parameters, and create a series of output files. Is this possible?===<br />
<br />
('''Updated for version 1.2''') We can change parameters that have an alias through the command line.<br />
<br />
For example, offscreen pipeline in offscreen.vt always creates the file called image.png. If you want generate it with a different filename:<br />
<br />
<code>python vistrails.py -b ../examples/offscreen.vt:offscreen -a"filename=other.png"</code><br />
<br />
<code>filename</code> in the example above is the alias name assigned to the parameter in the <code>value</code> method inside the String module. When running a pipeline from the command line, VisTrails will try to start the spreadsheet automatically if the pipeline requires it. For example, this other execution will also start the spreadsheet (attention to how $ characters are escaped when running on bash): <br />
<br />
<code>python vistrails.py -b ../examples/head.vt:aliases -a"isovalue=30\$&\$diffuse_color=0.8,0.4,0.2"</code><br />
<br />
You can also execute more than one pipeline on the command line:<br />
<br />
<code>python vistrails.py -b ../examples/head.vt:aliases ../examples/spx.vt:spx \<br />
-a"isovalue=30"</code><br />
<br />
Use the -a parameter only once regardless the number of pipelines. [http://www.vistrails.org/usersguide/v2.1/html/batch.html#running-a-workflow-with-specific-parameters Running a Workflow with Specific Parameters]<br />
<br />
=== I can load a vistrail, and the version tree shows up fine. However, no pipelines appear when I click on a version. What gives? ===<br />
<br />
The most likely reason is that the vistrail uses a package that is not registered with VisTrails. You need to identify the needed package and add it to your <code>.vistrails/startup.py</code>. A single line like the following should be enough:<br />
<br />
addPackage('enter_package_name_here')<br />
<br />
Some packages might need more information. For example:<br />
<br />
addPackage('afront', executable_path='/path/to/afront')<br />
<br />
Refer to the package documentation for details. The one inconvenient step is that currently there's no automated way to describe what is the missing package. We're working on this feature for future releases.<br />
<br />
=== I have a workflow that reads a file and then does some processing. The first time it runs, it executes correctly. But in subsequent, nothing happens. ===<br />
<br />
VisTrails caches by default, so after a workflow is executed, if none of its parameters change, it won't be executed again. <br />
<br />
If a workflow reads a file using the basic module File, VisTrails does check whether the file was modified since the last run. It does so by keeping a signature that is based on the modification time of the file. And if the file was modified, the File module and all downstream modules (the ones which depend on File) will be executed. <br />
<br />
<br />
''Note: If you would like your input and output data to be versioned, you can use the Persistence package.''<br />
<br />
If you do not want VisTrails to cache executions, you can turn off caching: go to Menu Edit -> Preferences and in the General Configuration tab, change Cache execution results to Never. [http://www.vistrails.org/usersguide/v2.1/html/getting_started.html#workflow-execution Workflow Execution]<br />
<br />
=== Can VisTrails execute workflows in parallel? ===<br />
<br />
The VisTrails server can only execute pipelines in parallel if there's more than one instance of VisTrails running. The command <br />
<br />
<code> self.rpcserver = ThreadedXMLRPCServer((self.temp_xml_rpc_options.server, self.temp_xml_rpc_options.port))</code><br />
<br />
starts a multithreaded version of the XML-RPC server, so it will create a thread for each request received by the server. The problem is that Qt/PyQT doesn't allow these multiple threads create GUI objects, only in the main thread. To overcome this limitation, the multithreaded version can instantiate other single threaded versions of VisTrails and put them in a queue, so workflow executions and other GUI-related requests, such as generating workflow graphs and history trees can be forwarded to this queue, and each instance takes turns in answering the request. If the results are in the cache, the multithreaded version answers the requests directly.<br />
<br />
Note that this infrastructure works on Linux only. To make this work on Windows, you have to create a script similar to start_vistrails_xvfb.sh (located in the scripts folder) where you can send the number of other instances via command-line options to VisTrails. The command line options are:<br />
<br />
python vistrails_server.py -T <ADDRESS> -R <PORT> -O<NUMBER_OF_OTHER_VISTRAILS_INSTANCES> [-M]&<br />
<br />
If you want the main vistrails instance to be multithreaded, use the -M at the end.<br />
<br />
After creating this script, update function start_other_instances in vistrails/gui/application_server.py lines 1007-1023 and set the script variable to point to your script. You may also have to change the arguments sent to your script (line 1016: for example, you don't need to set a virtual display). You will need to change the path to the stop_vistrails_server.py script (on line 1026) according to your installation path. [http://www.vistrails.org/usersguide/v2.1/html/batch.html#executing-workflows-in-parallel Executing Workflows in Parallel]<br />
<br />
=== When a workflow is executed, what do the colors mean? ===<br />
<br />
- lilac: module was notexecuted<br />
<br />
- yellow: module is currently being executed <br />
<br />
- green: module was successfully executed <br />
<br />
- orange: module was cached<br />
<br />
- red: the execution of the module failed<br />
<br />
[http://www.vistrails.org/usersguide/v2.1/html/getting_started.html#workflow-execution Workflow Execution]<br />
<br />
=== Workflow execution hangs on Windows ===<br />
<br />
This can happen if you are using "quick edit mode" in the console and have print statements in your code. Standard output can then get blocked by the console. Pressing space in the console resumes the execution. To avoid this problem, either disable "quick edit mode", or avoid print statements in your code.<br />
<br />
=== VisTrails do not install Missing System Packages ===<br />
<br />
If VisTrails do not try to install missing system packages it may be because it cannot determine your system type. I that case you can run this (in python) to determine your system type:<br />
<br />
import platform<br />
platform.linux_distribution()<br />
<br />
And add this system name to gui/bundles/utils.py by, e.g., modifying the _guess_ubuntu method (if your system is apt-based): <br />
<br />
def _guess_ubuntu():<br />
return platform.linux_distribution()[0]=='Ubuntu' or \<br />
platform.linux_distribution()[0]=='YourSystemName'<br />
<br />
=== Cannot update subworkflows after upgrading packages or vistrails version ===<br />
<br />
When packages used by a subworkflow is upgraded, any subworkflows that use it will be automatically upgraded. It may then lose the ability to be updated to a newer local subworkflow. In this case the subworkflow needs to be updated by hand by removing it from the pipeline and be dragged in again from the module palette. This may get fixed in a future release.<br />
<br />
== Building workflows ==<br />
<br />
=== Is there a way to give each widget a "display name" in addition to the module name at the center of the widget? ===<br />
<br />
Yes, a "display name" can be assigned to a module by selecting the triangle in its top right corner to open a popup menu and selecting the Set Module Label... menu item. You will then be prompted to enter the "display name". [http://www.vistrails.org/usersguide/v2.1/html/creating.html#configuring-module-labels Changing Module Labels]<br />
<br />
=== Is there a way to re-center the picture-in-picture (PiP) view? ===<br />
<br />
Yes. If you click on the PIP window to bring it to focus, you can press Ctrl-R (or Command-R on Mac) to re-center the PiP window. [http://www.vistrails.org/usersguide/v2.1/html/getting_started.html#vistrails-interaction Vistrails Interaction]<br />
<br />
=== How do I search for a literal "?" (question mark) in the search box in the Property panel? ===<br />
<br />
Since we allow regular expressions in our search box, question marks are treated as meta-characters. Thus, searching for "?" returns everything and "abc?" will return everything containing "abc". You need to use "\?" instead to search for "?". So the search for "??" would be "\?\?". [http://www.vistrails.org/usersguide/v2.1/html/querying.html#textual-queries Textual Queries]<br />
<br />
=== Saving a vistrail fails when Running VisTrails on Windows inside a Virtual Machine ===<br />
<br />
After installing Windows in a Virtual Machine, the path to zip.exe may be missing, and you may see this error when trying to save a vistrail:<br />
<br />
WindowsError: [Error 2] The system cannot find the file specified: '***/vt.zip'<br />
<br />
Then you need to add the path to zip.exe, which is included in the binary distribution of VisTrails, to your PATH variable.<br />
<br />
== Using VisTrails as a server ==<br />
<br />
=== What is the VisTrails server-mode? ===<br />
<br />
Using the VisTrails server mode, it is possible to ''execute workflows and control VisTrails through another application''. For example, the CrowdLabs Web portal (http://www.crowdlabs.org) accesses a VisTrails sever to execute workflows, retrieve and display vistrail trees and workflows. [http://www.vistrails.org/usersguide/v2.1/html/batch.html#using-vistrails-as-a-server Using VisTrails as a Server]<br />
<br />
=== How do I execute workflows and control VisTrails through another application? ===<br />
<br />
The way you access the server is by doing XML-RPC calls. In the current VisTrails release, we include a set of PHP scripts that can talk to a VisTrails server instance. They are in "extensions/http" folder. The files are reasonably well documented. Also, it should be not difficult to create python scripts to access the server (just use xmlrpclib module).<br />
<br />
Note that the VisTrails server requires the provenance and workflows to be in a database. More detailed instructions on how to setup the server and the database<br />
are available here:<br />
<br />
http://www.crowdlabs.org/site_media/static/dev_docs/vistrails_server_setup.html<br />
<br />
http://www.crowdlabs.org/site_media/static/dev_docs/vistrails_database_setup.html<br />
<br />
If what you want is just to execute a series of workflows in batch mode, a simpler solution would be to ''use the VisTrails client in batch mode''. Chapter 12 of the user's guide contains detailed information and examples on that. [http://www.vistrails.org/usersguide/v2.1/html/batch.html#sec-cli-batch Running VisTrails in Batch Mode]<br />
<br />
=== VisTrails server executes a workflow but generates a blank image and generates the error message ''cannot get access to X server'' ===<br />
<br />
You will need to check if the display the server is trying to use is a valid display (by default it uses the display 0). On linux, the command ''w'' will list the logged users and the display associated with them (''FROM'' column).<br />
<br />
Note that the VisTrails server requires the machine to be running X.<br />
<br />
=== cannot get access to X server ===<br />
<br />
Running VisTrails in '''server or batch mode''' requires a connection to an X server. <br />
<br />
No additional setup is required if you run VisTrails on a terminal because you are already logged in to X. To make it work in other scenarios, you need to run the python command through Xvfb or make sure you can run cgi scripts that access the GUI.<br />
<br />
If you can run Xvfb, you can use the following script, where you need toconfigure the first four variables according to your system: http://www.vistrails.org/images/Run_vistrails_batch_xvfb.script.sh.txt<br />
<br />
(To run the script, rename the file and remove the ".txt")<br />
<br />
<br />
You should also modify yout cgi script to invoke the bash script instead of vistrails directly. The bash script will accept the virtual display, the vistrail file and workflow tag as input arguments.<br />
<br />
Another possibility is if your workflow does not require the GUI, you can use VisTrails as a regular python module and it will not require the GUI or X Server to run. This functionality is available in the nightly builds and will be included in VisTrails 2.0 beta to be released soon. There is an example of how to use this feature in our FAQ: http://www.vistrails.org/index.php/FAQ#Using_VisTrails_as_a_Python_module<br />
<br />
== Problems starting VisTrails ==<br />
<br />
=== Setup was unable to create the directory "N:\.vistrails" ===<br />
<br />
When VisTrails is installing, it tries to create the .vistrails folder in the users %HOMEPATH% directory. In some Windows installations, network accounts are set to a directory that a user does not have write access to. Consequently, the installation will fail. To get around this problem, you can use the "-S <directory>" flag when starting VisTrails. This option allows you to put the .vistrails directory wherever you wish. You could also write a short script that automatically invokes VisTrails with the "-S" flag pointing to a directory that makes sense to your network. If you are unable to install VisTrails, you can run the installer after setting a new home path from the command line like this:<br />
<br />
set HOMEPATH=\My\New\Home\<br />
set HOMEDRIVE=C:<br />
vistrails-setup-2.0.1-xxx.exe<br />
<br />
== Using VisTrails as a Python module ==<br />
<br />
=== Can I use VisTrails as a Python module without installing PyQt? ===<br />
<br />
Yes! We have improved the ability to use VisTrails from other software, and have eliminated most GUI (PyQt) dependencies in the core part of the code. Thus, you can now work with workflow versions and provenance information in a standard python shell. Note packages that directly rely on the GUI like the VisTrails Spreadsheet will still require PyQt to be installed.<br />
<br />
=== How do I open and execute workflows in a standard python shell? ===<br />
<br />
Here is a simple example that shows how you can open and execute a workflow from a Python script:<br />
<br />
>>> import vistrails as vt<br />
>>><br />
>>> vistrail = vt.load_vistrail('simplemath.vt')<br />
>>> vistrail.select_latest_version()<br />
>>> result = vistrail.execute(in_a=2, in_b=4)<br />
>>> result.output_port('out_plus')<br />
6.0<br />
<br />
A more complete example is available in the VisTrails distribution as [https://github.com/VisTrails/VisTrails/blob/v2.2/examples/api/ipython-notebook.ipynb examples/api/ipython-notebook.ipynb]<br />
<br />
== Control Flow ==<br />
<br />
=== Note: using map ===<br />
<br />
When using 'map', the module (or subworkflow) used as function port in the map module MUST be a function, i.e., it can only define 1 output port. [http://www.vistrails.org/usersguide/v2.1/html/controlflow.html#the-map-operator The Map Operator]<br />
<br />
== Spreadsheet ==<br />
<br />
Where ''pipeline'' is a version number or a tag.<br />
<br />
=== How can I save an image from the spreadsheet? ===<br />
<br />
While having the focus on a spreadsheet cell and select the camera on the toolbar to take a snapshot. The system will prompt you for the location and file name where it should be saved. The other icons can be used for saving multiple images that can be used for generating an animation on demand. A whole sheet can also be saved by selecting Export (either from the menu or from the toolbar). [http://www.vistrails.org/usersguide/v2.1/html/spreadsheet.html#saving-a-spreadsheet-image Saving a Spreadsheet Image]<br />
<br />
===Is it possible to save the complete state of the spreadsheet?===<br />
[http://www.vistrails.org/usersguide/v2.1/html/spreadsheet.html#saving-a-spreadsheet Saving a Spreadsheet]<br />
<br />
=== Can I view multiple sheets at the same time? ===<br />
<br />
Yes. Each sheet on the spreadsheet can be displayed as a dock widget separated from the main spreadsheet window by dragging its tab name out of the tab bar at the bottom of the spreadsheet. [http://www.vistrails.org/usersguide/v2.1/html/spreadsheet.html#multiple-spreadsheets Multiple Spreadsheets]<br />
<br />
=== Then, how can I put back a separated sheet? ===<br />
<br />
A sheet can be docked back to the main window by dragging it back to the tab bar or double-click on its title bar. [http://www.vistrails.org/usersguide/v2.1/html/spreadsheet.html#multiple-spreadsheets Multiple Spreadsheets]<br />
<br />
=== How can I order sheets on the spreadsheet? ===<br />
<br />
This can be done by dragging the sheet name on the bottom top bar and drop it to the right place. [http://www.vistrails.org/usersguide/v2.1/html/spreadsheet.html#multiple-spreadsheets Multiple Spreadsheets]<br />
<br />
=== Can I control where a cell will be placed on the spreadsheet window? ===<br />
<br />
By default, an unoccupied cell on the active sheet will be chosen to display the result. However, you can specify exactly in the pipeline where a spreadsheet cell will be placed by using CellLocation and SheetReference. CellLocation specifies the location (row and column) of a cell when connecting to a spreadsheet cell (VTKCell, ImageViewerCell, ...). Similarly, a SheetReference module (when connecting to a CellLocation) will specify which sheet the cell will be put on given its name, minimum row size and minimum column size. There is an example of this in examples/vtk.xml (select the version below Double Renderer). [http://www.vistrails.org/usersguide/v2.1/html/spreadsheet.html#sending-output-to-the-spreadsheet Sending Output to the Spreadsheet]<br />
<br />
=== How do I output results to the spreadsheet? ===<br />
<br />
By inspecting the VisTrails Spreadsheet package (in the list of packages, to the left of the pipeline builder), you<br />
can see there are built-in cells for different kinds of data, e.g., RichTextCell to display HTML and plain text. op<br />
You (the user) can also define new cell types to display application-specific data. For example, we have developed<br />
VtkCell, MplFigureCell, and OpenGLCell. It is possible to display pretty much anything on the Spreadsheet! [http://www.vistrails.org/usersguide/v2.1/html/spreadsheet.html#sending-output-to-the-spreadsheet Sending Output to the Spreadsheet]<br />
<br />
Examples of writing cell modules can be found in:<br />
RichTextCell: packages/spreadsheet/widgets/richtext/richtext.py<br />
VTK: packages/vtk/vtkcell.py<br />
<br />
Here is the summary of some requirements on a cell widget:<br />
<br />
(1) It must be a Qt widget. It should inherit from spreadsheet_cell.QCellWidget in the spreadsheet package. Although any Qt Widget would work, certain features such as animation will not be available (without rewriting it).<br />
<br />
(2) It must re-implement the updateContents() function to take a set of inputs (usually coming from input ports of a wrapper Module) and display on the cells. VisTrails uses this function to update/reuse cells on the spreadsheet when new data comes in.<br />
<br />
(3) It needs a wrapper VisTrails Module (inherited from basic_widgets.SpreadsheetCell of the spreadsheet package). Inside the compute() method of this module, it may call self.display(CellWidgetType, (inputs)) to trigger the display event on the spreadsheet. [http://www.vistrails.org/usersguide/v2.1/html/spreadsheet.html#advanced-cell-options Advanced Cell Options]<br />
<br />
=== How do I control the default number of cells in the spreadsheet? ===<br />
<br />
You can configure the rowCount and colCount using the preferences dialog. Just go to the Module Packages tab, select spreadsheet in the "Enabled packages" and press the Configure button. Then a list of all the configuration options for the spreadsheet will show up. [http://www.vistrails.org/usersguide/v2.1/html/spreadsheet.html#custom-layout-options Custom Layout Options]<br />
<br />
=== Is it possible to launch a web browser from the vistrails spreadsheet? We would like to output several urls from a parameter sweep and then have the option to click on each one to view the resulting page. I can view the page within the spreadsheet, but it is really too crowded.===<br />
<br />
Currently, there isn't a widget that provides exactly this functionality, but I can think of a few solutions that may work for you:<br />
<br />
(1) You can use parameter exploration to generate multiple sheets so you might have an exploration that opens each page in a new sheet. Use the third column/dimension in the exploration interface to have a parameter span sheets.<br />
<br />
(2) The spreadsheet is extensible so you can write a custom spreadsheet cell widget that has a button or label with the desired link (a QLabel with openExternalLinks set to True, for example).<br />
<br />
(3) You can tweak the existing RichTextCell be adding the line "self.browser.setOpenExternalLinks(True)" at line 63 of the source file "vistrails/packages/spreadsheet/widgets/richtext/richtext.py". Then, if your workflow creates a file with html markup text like "<a href="http://www.vistrails.org/">VisTrails</a>" connected to a RichTextCell, clicking on the rendered link in the cell will open it in a web browser. You need to add the aforementioned line to the source to let Qt know that you want the link opened externally; by default, it will just issue an event that isn't processed. [http://www.vistrails.org/usersguide/v2.1/html/spreadsheet.html#launching-a-web-browser Launching a Web Browser]<br />
<br />
== Integrating your software into VisTrails ==<br />
<br />
===How can I integrate my own program into VisTrails?===<br />
<br />
The easiest way is to create a package. Writing a package is often very simple; please refer to [http://www.vistrails.org/usersguide/v2.1/html/packages.html this section of the users' guide].<br />
<br />
You can also dynamically generate modules. For an example see:<br />
<br />
[http://www.vistrails.org/usersguide/v2.1/html/packages.html#generating-modules-dynamically Generating Modules Dynamically]<br />
<br />
In particular, see the new_module call which uses python's type() function to generate new classes dynamically.<br />
<br />
===How do I add a port that is not visible on the module (when it appears on the design canvas)?===<br />
<br />
This can be accomplished via the "optional" argument. This is the fourth argument of add_input_port (add_output_port) or can be specified as a kwarg. In your example, this would look like:<br />
<br />
reg.add_input_port(MyModule, "MyPort", (core.modules.basic_modules.String, 'MyPort Name'), True)<br />
<br />
or with kwargs<br />
<br />
reg.add_input_port(MyModule, "MyPort", (core.modules.basic_modules.String, 'MyPort Name'),\<br />
optional=True)<br />
<br />
or<br />
<br />
_input_ports = [('MyPort', '(core.modules.basic_modules.String)', {"optional": True})]<br />
[http://www.vistrails.org/usersguide/v2.1/html/packages.html#configuring-ports Configuring Ports]<br />
<br />
=== How do modules deal with multiple inputs in a same port? ===<br />
<br />
(And should that even be allowed?)<br />
<br />
For compatibility reasons, we do need to allow multiple connections to an input port. However, most package developers should never have to use this, and so we do our best to hide it. the default behavior for getting inputs from a port, then, is to always return a single input.<br />
<br />
If on your module you need multiple inputs connected to a single port, use the 'forceGetInputListFromPort' method. It will return a list of all the data items coming through the port. The spreadsheet package uses this feature, so look there for usage examples (vistrails/packages/spreadsheet/basic_widgets.py)<br />
[http://www.vistrails.org/usersguide/v2.1/html/packages.html#configuring-ports Configuring Ports]<br />
<br />
=== Are there mechanisms for attaching widgets to different modules/parameters? ===<br />
<br />
Right now, we have a mechanism for putting a specific widget for an input port. For example, if a port is SetColor(red, green, blue), we can put a color wheel widget there. Or we can also replace the SetFileName port with a File Widget. However, this is not per parameter (only per port). We are currently working on this problem.<br />
<br />
=== Can I organize my package so it appears hierarchical in the module palette? ===<br />
<br />
Yes. Use the namespace keyword argument when adding the module to the registry. For example,<br />
<br />
registry.add_module(MyModule, namespace='MyNamespace')<br />
[http://www.vistrails.org/usersguide/v2.1/html/packages.html#configuring-modules Configuring Modules - Hierarchy and Visibility]<br />
<br />
=== Can I nest namespaces? ===<br />
<br />
Yes. Use the '|' character to separate different the hierarchy. For example,<br />
<br />
registry.add_module(MyModule, namespace='ParentNamespace|ChildNamespace')<br />
<br />
[http://www.vistrails.org/usersguide/v2.1/html/packages.html#configuring-modules Configuring Modules - Hierarchy and Visibility]<br />
<br />
=== Are there shortcuts for registry initialization? ===<br />
<br />
Yes. If you define _modules as a list of classes in the __init__.py file of your package, VisTrails will attempt to load all classes specified as modules. You can provide add_module options as keyword arguments by specifying a tuple (class, kwargs) in the list. For example:<br />
<br />
_modules = [MyModule1, (MyModule2, {'namespace': 'MyNamespace'})]<br />
<br />
In addition, you need to identify the ports of your modules as a field in your class by defining _input_ports and _output_ports lists. Here, the items in each list must be tuples of the form (portName, portSignature, optional=False, sort_key=-1). For example:<br />
<br />
class MyModule(Module):<br />
def compute(self):<br />
pass<br />
<br />
_input_ports = [('firstInput', String), ('secondInput', Integer, True)]<br />
_output_ports = [('firstOutput', String), ('secondOutput', String)]<br />
<br />
[http://www.vistrails.org/usersguide/v2.1/html/packages.html#customizing-modules-and-ports Customizing Modules and Ports]<br />
<br />
=== Can I define ports to be of types that I do not import into my package? ===<br />
<br />
Yes. You can pass an identifier string as the portSignature instead. The port_signature string is defined by:<br />
<br />
<module_string> := <package_identifier>:[<namespace>|]<module_name>,<br />
<port_signature> := (<module_string>*)<br />
<br />
For example,<br />
<br />
registry.add_input_port(MyModule, 'myInputPort', '(edu.utah.sci.vistrails.basic:String)')<br />
<br />
or<br />
<br />
_input_ports = [('myInputPort', '(edu.utah.sci.vistrails.basic:String)')]<br />
<br />
[http://www.vistrails.org/usersguide/v2.1/html/packages.html#configuring-ports Configuring Ports - Port Types]<br />
<br />
=== What do I need to change in my package to make it reloadable (new in v1.4.2)? ===<br />
See [http://www.vistrails.org/usersguide/v2.1/html/packages.html#creating-reloadable-packages Creating Reloadable Packages] for an explanation.<br />
<br />
=== Can I add default values or labels for parameters? ===<br />
<br />
Yes. Versions 1.4 and greater support these features. See [http://www.vistrails.org/usersguide/v2.1/html/packages.html#configuring-ports Configuring Ports - Default Values and Labels] for more details.<br />
<br />
=== How can I access the default values for a parameter? ===<br />
<br />
The default values are stored in PortSpec.defaults for each port.<br />
<br />
=== I want to write a module to load HDF data whose output (e.g., data, string) varies according to the input I give it. Is is possible to do this in VisTrails, and if yes, how can I do that? Ideally, I would like to avoid having to change the connection of my output every time I change the input. ===<br />
<br />
There are a few ways to tackle this - each has it's own benefits and pitfalls. Firstly, module connections do respect class hierarchies as we're familiar with in object oriented languages. For instance, A module can output a Constant of which String, Float, Integer, etc are specifications. In this way, you can have a subclass of something like HDFData be passed out of the module and the connections will be established regardless of the sub-type. This is a bit dangerous though. Modules downstream of such a class may not really know how to operate on certain types derived from the super-class. Extreme care must be taken both when creating the modules as well as connecting them to prevent things like this from happening.<br />
<br />
A second method that I employ in several different packages is the idea of a container class. For instance, the NumSciPy package uses a relatively generic container "Numpy Array" to encapsulate the data. Of course, these encapsulating objects can store dictionaries that other modules can easily access and understand how to operate on. Although this method is slightly more work, the benefits of a stricter typing of ports is beneficial - particularly upon interfacing with other packages that may depend on strongly typed constants (for example).<br />
[http://www.vistrails.org/usersguide/v2.1/html/packages.html#configuring-ports Varying Output According to the Input]<br />
<br />
=== I need to determine, at run-time, whether or not a "child" module is attached to the output port of a "parent" module. (I do not specifically need to know which child; just if there is one). ===<br />
<br />
The outputPorts dictionary of the base Module stores this information. Thus, you should be able to check<br />
<br />
("myPortName" in self.outputPorts)<br />
<br />
on the parent module to check if there are any downstream connections from the port "myPortName". This might be used, for example, to only set results for output ports that will be used. ***Note***, however, that the caching algorithm assumes that all outputs are set so adding a new connection to a previously unconnected output port will not work as desired if that module is cached. For this reason, I would currently recommend making such a module not cacheable. Another possibility is overriding the update() method to check the outputPorts and set the upToDate flag if they are not equal. In a single, limited test, this seemed to work, but be warned that it is not fully tested. Here is an example:<br />
<br />
class TestModule(Module):<br />
_output_ports = [('a1', '(edu.utah.sci.vistrails.basic:String)'),<br />
('a2', '(edu.utah.sci.vistrails.basic:String)')]<br />
def __init__(self):<br />
Module.__init__(self)<br />
self._cached_output_ports = set()<br />
<br />
def update(self):<br />
if len(set(self.outputPorts) - self._cached_output_ports) > 0:<br />
self.upToDate = False<br />
Module.update(self)<br />
<br />
def compute(self):<br />
if "a1" in self.outputPorts:<br />
self.setResult("a1", "test")<br />
if "a2" in self.outputPorts:<br />
self.setResult("a2", "test2")<br />
self._cached_output_ports = set(self.outputPorts)<br />
<br />
[http://www.vistrails.org/usersguide/v2.1/html/packages.html#configuring-ports Configuring Ports - Determining Whether or Not a Module is Attached to an Output Port]<br />
<br />
=== How can I make a module not display in the modules list? ===<br />
<br />
You should set the abstract parameter to True when adding the module to the registry. Using the original syntax, this looks like:<br />
<br />
def initialize():<br />
reg = core.modules.module_registry.get_module_registry()<br />
reg.add_module(InvisibleModule, abstract=True)<br />
# ...<br />
<br />
With the _modules dictionary shortcut (for more details, see [http://www.vistrails.org/index.php/FAQ#Are_there_shortcuts_for_registry_initialization.3F the FAQ section on this]), you include it in a kwargs dict as part of a module tuple:<br />
<br />
_modules = [AnotherModule, (InvisibleModule, {'abstract': True})]<br />
<br />
There is also a 'hide_descriptor' parameter that prevents the module from appearing in the module palette without declaring it to be abstract.<br />
<br />
The technical difference between the two is that 'abstract' will not add the item to the module palette while 'hide_descriptor' does add the item but immediately hides it. If the module should never be instantiated in a workflow, declare it abstract. If you don't want users to be able to add the module to a pipeline, but you have code that may add it programmatically, declare it with hide_descriptor=True.<br />
<br />
[http://www.vistrails.org/usersguide/v2.1/html/packages.html#configuring-modules Configuring Modules - Hierarchy and Visibility]<br />
<br />
=== How do I document individual ports? ===<br />
<br />
To access port documentation, users can right-click on the port in the port list and choose the corresponding menu item. To provide this documentation, you should define the <code>provide_input_port_documentation</code> and/or the <code>provide_output_port_documentation</code> '''class''' methods. Note that these methods take the class and the port name as arguments. For example,<br />
<br />
class MyModule(Module):<br />
_input_ports = [('test', '(edu.utah.sci.vistrails.basic:String)'),<br />
('test2', '(edu.utah.sci.vistrails.basic:String)')]<br />
port_docs = {'test': 'Some documentation',<br />
'test2': 'More documentation'}<br />
@classmethod<br />
def provide_input_port_documentation(cls, port_name):<br />
return cls.port_docs[port_name]<br />
<br />
=== How do I access modules from other packages? ===<br />
<br />
Currently, it is best to access modules from the registry. First, make sure that any dependencies on another package are specified in <code>package_dependencies</code> method in <code>__init__.py</code>. To create a module from another package as an output, you can generate it from the registry. For example,<br />
<br />
from core.modules.module_registry import get_module_registry<br />
from core.modules.vistrails_module import Module<br />
<br />
class ReturnFigManager(Module):<br />
_output_ports = [('figManager', <br />
'(edu.utah.sci.vistrails.matplotlib:MplFigureManager)')]<br />
def compute(self):<br />
reg = get_module_registry()<br />
wrapper = \<br />
reg.get_descriptor_by_name("edu.utah.sci.vistrails.matplotlib", <br />
"MplFigureManager").module()<br />
wrapper.figManager = "blah"<br />
self.setResult('figManager', wrapper)<br />
<br />
You can also create subclass from classes obtained from the registry. For example,<br />
<br />
MplFigureManager = get_module_registry().get_descriptor_by_name(<br />
"edu.utah.sci.vistrails.matplotlib", <br />
"MplFigureManager").module<br />
class MplFigureManagerSubclass(MplFigureManager):<br />
pass<br />
<br />
<br />
=== How do I create a custom module configuration widget? ===<br />
<br />
See [[Module Configuration Example]] for a full example and notes about doing this.<br />
<br />
=== Can I make a PythonSource module cacheable? ===<br />
<br />
Yes. If you have a module that you are planning to re-use in a workflow, we recommend making a packaged module (which are by default cacheable). However, you can make a PythonSource (which are by default not cacheable) cacheable using the line<br />
<br />
self.is_cacheable = lambda *args, **kwargs: True<br />
<br />
in the source of the PythonSource module.<br />
<br />
== The Console ==<br />
<br />
=== Where should I go to find out what I can call from the console and how to import it? ===<br />
<br />
We have tried to make some methods more accessible in the console via an api. You can import the api via <code>from vistrails import api</code> in the console and see the available methods with <code>dir(api)</code>. To open a vistrail:<br />
<br />
from vistrails import api<br />
api.open_vistrail_from_file('/Applications/VisTrails/examples/terminator.vt')<br />
<br />
To execute a version of a workflow, you currently have to go through the controller:<br />
<br />
api.select_version('Histogram')<br />
api.get_current_controller().execute_current_workflow()<br />
<br />
Currently, only a subset of VisTrails functionality is directly available from the api. However, since VisTrails is written in python, you can dig down starting with the VistrailsApplication or controller object to expose most of our internal methods. If you have suggestions for calls to be added to the api, please let us know.<br />
<br />
One other feature that we're working on, but is still in progress is the ability to construct workflows via the console. For example:<br />
<br />
vtk = load_package('edu.utah.sci.vistrails.vtk')<br />
vtk.vtkDataSetReader() # adds a vtkDataSetReader module to the pipeline<br />
# click on the new module<br />
a = selected_modules()[0] # get the one currently selected module<br />
a.SetFile('/vistrails/examples/data/head120.vtk') # sets the SetFile parmaeter for the data set reader<br />
b = vtk.vtkContourFilter() # adds a vtkContourFilter module to the pipeline and saves to var b<br />
b.SetInputConnection0(a.GetOutputPort0()) # connects a's GetOutputPort0 port to b's SetInputConnection0<br />
[http://www.vistrails.org/usersguide/v2.1/html/batch.html#finding-methods-via-the-command-line Finding Methods Via the Command Line]<br />
<br />
== Persistence Package ==<br />
<br />
=== How do I use the output of one workflow as the input for another using the persistence package? ===<br />
<br />
You need to configure the persistence modules using the module's configuration dialog. After adding a <code>PersistentOutputFile</code> to the workflow, click on the triangle in the upper-right corner of the <code>PersistentOutputFile</code>, and select "Edit Configuration" from the menu that appears. In this dialog, select "Create New Reference" and give the reference a name (and any space-delimited tags). Upon running that workflow, the data will be written to the persistent store. In the second workflow where you wish to use that file, add a <code>PersistentInputFile</code> and go to its configuration dialog in the same manner as with the output file. In that dialog, select "Use Existing Reference" and select the data that you just added in the first workflow in the list of files below. Now, when you run that workflow, it will grab the data from the persistent store.<br />
<br />
Here is an example: [[Media:offscreen_persistent.vt | offscreen_persistent.vt]]. Run the "persistent offscreen" workflow first, and then run the "display persistent output" to use the output of the first workflow as the input for the second.<br />
<br />
== VTK ==<br />
<br />
=== Given a VTK visualization, how can I generate a webpage from it? ===<br />
<br />
Check out the html pipeline in offscreen.xml.<br />
<br />
=== I'm trying to use VTK, but there doesn't seem to be any output. What is wrong? ===<br />
<br />
To use VTK on VisTrails, you need a slightly different way of connecting the renderer modules. Instead of using the standard RenderWindow/RenderWindowInteractor infrastructure, you simply connect the renderer to a VTKCell. The examples directory in the distribution has several VTK examples that illustrate.<br />
<br />
=== I am trying to add a module to the workflow via Python, but how can I access vtk modules? ===<br />
<br />
Here's an example:<br />
<br />
import api<br />
<br />
vtvtk = 'edu.utah.sci.vistrails.vtk'<br />
<br />
module = api.add_module(0, 0, vtvtk, 'vtkContourFilter', '')<br />
<br />
<br />
The third argument in add_module is the package identifier. You can find this in the "Module Packages" panel of the Preferences; just click on the package you're interested in and it will appear in the information on the right.<br />
<br />
== matplotlib ==<br />
<br />
=== I'm experiencing a problem with Latex labels and the matplotlib that comes with VisTrails 1.5. The script below entered to the interpreter that comes with VT is sufficient to reproduce it. ===<br />
<br />
import matplotlib.pyplot as plt<br />
plt.plot([1,2,3],[1,2,3])<br />
plt.xlabel("$foo$")<br />
<br />
Remove your ~/.matplotlib folder and re-start VisTrails<br />
<br />
== rpy ==<br />
<br />
=== Package rpy fails with "module object has no attribute RVector" ===<br />
<br />
The rpy package needs to be updated to support a newer rpy version. In "packages/rpy/init.py", replace all instances of "objects.RVector" with "objects.Vector", or use [https://raw.github.com/VisTrails/VisTrails/d0711a8d5ba8992662f69ed7924c625a3658b6a7/vistrails/packages/rpy/init.py this] file. <br />
<br />
== JobSubmission ==<br />
<br />
The JobSubmission package depends on the stable version of BatchQ. Download https://github.com/troelsfr/BatchQ/archive/stable.zip, copy the "BatchQ-stable/batchq" directory to your local site-packages folder. Copying it to the "vistrails/packages/JobSubmission" folder should also work. See batchq/contrib/vistrails for examples.<br />
<br />
== VisTrails Development ==<br />
<br />
=== I would like to build VisTrails from source. Are there instructions on how to do this? ===<br />
<br />
Yes! Take a look at [http://www.vistrails.org/usersguide/v2.1/html/getting_started.html#installing-vistrails-from-source Installing VisTrails from source]<br />
<br />
== Accessing Provenance Information ==<br />
<br />
=== How do I access the information in the execution log? ===<br />
<br />
The code responsible for storing execution information is located in the "core/log" directories, and the code that generates much of that information is in "core/interpreter/cached.py". Modules can add execution-specific annotations to provenance via annotate() calls during execution, but much of the data (like timing and errors) is captured by the LogController and CachedInterpreter (the execution engine) objects. To analyze the log from a vistrail (.vt) file, you might have something like the following:<br />
<br />
<code><br />
import core.log.log<br />
import db.services.io<br />
<br />
def run(fname):<br />
# open the .vt bundle specified by the filename "fname"<br />
bundle = db.services.io.open_vistrail_bundle_from_zip_xml(fname)[0]<br />
# get the log filename<br />
log_fname = bundle.vistrail.db_log_filename<br />
if log_fname is not None:<br />
# open the log<br />
log = db.services.io.open_log_from_xml(log_fname, True)<br />
# convert the log from a db object<br />
core.log.log.Log.convert(log)<br />
for workflow_exec in log.workflow_execs:<br />
print 'workflow version:', workflow_exec.parent_version<br />
print 'time started:', workflow_exec.ts_start<br />
print 'time ended:', workflow_exec.ts_end<br />
print 'modules executed:', [i.module_id <br />
for i in workflow_exec.item_execs]<br />
if __name__ == '__main__':<br />
run("some_vistrail.vt")<br />
</code><br />
<br />
You should be able to see what information is available by looking at the "core/log" classes. [http://www.vistrails.org/usersguide/v2.1/html/log.html Accessing the Execution Log]<br />
<br />
== VisTrails Binaries ==<br />
=== Is there a Mac OS X 10.6+ x64 binary of the version 1.7 of VisTrails available? ===<br />
<br />
We don't have a 64bit Mac binary for v1.7 release because at the time we didn't have 64 bit versions of the libraries shipped in the 1.7 binary.<br />
<br />
However, it is possible to update a 64bit or any other binary with a source release of VisTrails, including the sources of 1.7 version or the nightly builds.<br />
<br />
Assuming you have the sources of 1.7 in /vistrails1.7 and the 64bit binary in /Applications/VisTrails1.7 do the following steps:<br />
cp /vistrails1.7/vistrails/vistrails.py /Applications/VisTrails1.7/VisTrails.app/Contents/Resources<br />
cp -r /vistrails1.7/vistrails/api /Applications/VisTrails1.7/VisTrails.app/Contents/Resources/lib/python2.7/<br />
cp -r /vistrails1.7/vistrails/core /Applications/VisTrails1.7/VisTrails.app/Contents/Resources/lib/python2.7/<br />
cp -r /vistrails1.7/vistrails/db /Applications/VisTrails1.7/VisTrails.app/Contents/Resources/lib/python2.7/<br />
cp -r /vistrails1.7/vistrails/gui /Applications/VisTrails1.7/VisTrails.app/Contents/Resources/lib/python2.7/<br />
cp -r /vistrails1.7/vistrails/packages /Applications/VisTrails1.7/VisTrails.app/Contents/Resources/lib/python2.7/<br />
cp -r /vistrails1.7/examples /Applications/VisTrails1.7/<br />
cp -r /vistrails1.7/extensions /Applications/VisTrails1.7/<br />
cp -r /vistrails1.7/scripts /Applications/VisTrails1.7/<br />
<br />
=== AVG Antivirus falsely report a virus in VisTrails 2.0.2 32-bit Windows installer ===<br />
<br />
Problematic file is vistrails/Python27/Lib/site-packages/_mysql.pyd:<br />
https://www.virustotal.com/en/file/d8aabd921b5eba8aabcce936ce3b92e3d1de43eb44c43d921ca1b9ab91d7fd81/analysis/1366640335/. This is most likely a false positive and can be ignored.<br />
<br />
=== VisTrails fails after upgrading to OSX 10.9 ===<br />
<br />
Reinstalling [http://xquartz.macosforge.org/ XQuartz] should solve the problem.</div>Remihttps://www.vistrails.org//index.php?title=FAQ&diff=13970FAQ2017-05-26T13:57:54Z<p>Remi: Update API docs</p>
<hr />
<div>Also check our [[known_issues|Known Issues]] page for troubleshooting.<br />
<br />
----<br />
== Running workflows ==<br />
<br />
=== How can I run a workflow using the command line? ===<br />
<br />
('''Updated for version 1.2''') Call vistrails using the following options:<br />
<br />
<code>python vistrails.py -b path_to_vistrails_file:pipeline </code><br />
<br />
where pipeline can be a version '''tag name''' or version '''id'''<br />
<br />
'''NOTE:''' If you downloaded the MacOS X bundle, you can run vistrails from the command line via the following commands in the Terminal. Change the current directory to wherever VisTrails was installed (often <code>/Applications</code>), and then type:<br />
<br />
<code>Vistrails.app/Contents/MacOS/vistrails [<cmd_line_options>]</code> [http://www.vistrails.org/usersguide/v2.1/html/batch.html#running-a-specific-workflow-in-batch-mode Running a Specific Workflow in Batch Mode]<br />
<br />
=== Using the command line, we'd like to execute a workflow multiple times, with slightly different parameters, and create a series of output files. Is this possible?===<br />
<br />
('''Updated for version 1.2''') We can change parameters that have an alias through the command line.<br />
<br />
For example, offscreen pipeline in offscreen.vt always creates the file called image.png. If you want generate it with a different filename:<br />
<br />
<code>python vistrails.py -b ../examples/offscreen.vt:offscreen -a"filename=other.png"</code><br />
<br />
<code>filename</code> in the example above is the alias name assigned to the parameter in the <code>value</code> method inside the String module. When running a pipeline from the command line, VisTrails will try to start the spreadsheet automatically if the pipeline requires it. For example, this other execution will also start the spreadsheet (attention to how $ characters are escaped when running on bash): <br />
<br />
<code>python vistrails.py -b ../examples/head.vt:aliases -a"isovalue=30\$&\$diffuse_color=0.8,0.4,0.2"</code><br />
<br />
You can also execute more than one pipeline on the command line:<br />
<br />
<code>python vistrails.py -b ../examples/head.vt:aliases ../examples/spx.vt:spx \<br />
-a"isovalue=30"</code><br />
<br />
Use the -a parameter only once regardless the number of pipelines. [http://www.vistrails.org/usersguide/v2.1/html/batch.html#running-a-workflow-with-specific-parameters Running a Workflow with Specific Parameters]<br />
<br />
=== I can load a vistrail, and the version tree shows up fine. However, no pipelines appear when I click on a version. What gives? ===<br />
<br />
The most likely reason is that the vistrail uses a package that is not registered with VisTrails. You need to identify the needed package and add it to your <code>.vistrails/startup.py</code>. A single line like the following should be enough:<br />
<br />
addPackage('enter_package_name_here')<br />
<br />
Some packages might need more information. For example:<br />
<br />
addPackage('afront', executable_path='/path/to/afront')<br />
<br />
Refer to the package documentation for details. The one inconvenient step is that currently there's no automated way to describe what is the missing package. We're working on this feature for future releases.<br />
<br />
=== I have a workflow that reads a file and then does some processing. The first time it runs, it executes correctly. But in subsequent, nothing happens. ===<br />
<br />
VisTrails caches by default, so after a workflow is executed, if none of its parameters change, it won't be executed again. <br />
<br />
If a workflow reads a file using the basic module File, VisTrails does check whether the file was modified since the last run. It does so by keeping a signature that is based on the modification time of the file. And if the file was modified, the File module and all downstream modules (the ones which depend on File) will be executed. <br />
<br />
<br />
''Note: If you would like your input and output data to be versioned, you can use the Persistence package.''<br />
<br />
If you do not want VisTrails to cache executions, you can turn off caching: go to Menu Edit -> Preferences and in the General Configuration tab, change Cache execution results to Never. [http://www.vistrails.org/usersguide/v2.1/html/getting_started.html#workflow-execution Workflow Execution]<br />
<br />
=== Can VisTrails execute workflows in parallel? ===<br />
<br />
The VisTrails server can only execute pipelines in parallel if there's more than one instance of VisTrails running. The command <br />
<br />
<code> self.rpcserver = ThreadedXMLRPCServer((self.temp_xml_rpc_options.server, self.temp_xml_rpc_options.port))</code><br />
<br />
starts a multithreaded version of the XML-RPC server, so it will create a thread for each request received by the server. The problem is that Qt/PyQT doesn't allow these multiple threads create GUI objects, only in the main thread. To overcome this limitation, the multithreaded version can instantiate other single threaded versions of VisTrails and put them in a queue, so workflow executions and other GUI-related requests, such as generating workflow graphs and history trees can be forwarded to this queue, and each instance takes turns in answering the request. If the results are in the cache, the multithreaded version answers the requests directly.<br />
<br />
Note that this infrastructure works on Linux only. To make this work on Windows, you have to create a script similar to start_vistrails_xvfb.sh (located in the scripts folder) where you can send the number of other instances via command-line options to VisTrails. The command line options are:<br />
<br />
python vistrails_server.py -T <ADDRESS> -R <PORT> -O<NUMBER_OF_OTHER_VISTRAILS_INSTANCES> [-M]&<br />
<br />
If you want the main vistrails instance to be multithreaded, use the -M at the end.<br />
<br />
After creating this script, update function start_other_instances in vistrails/gui/application_server.py lines 1007-1023 and set the script variable to point to your script. You may also have to change the arguments sent to your script (line 1016: for example, you don't need to set a virtual display). You will need to change the path to the stop_vistrails_server.py script (on line 1026) according to your installation path. [http://www.vistrails.org/usersguide/v2.1/html/batch.html#executing-workflows-in-parallel Executing Workflows in Parallel]<br />
<br />
=== When a workflow is executed, what do the colors mean? ===<br />
<br />
- lilac: module was notexecuted<br />
<br />
- yellow: module is currently being executed <br />
<br />
- green: module was successfully executed <br />
<br />
- orange: module was cached<br />
<br />
- red: the execution of the module failed<br />
<br />
[http://www.vistrails.org/usersguide/v2.1/html/getting_started.html#workflow-execution Workflow Execution]<br />
<br />
=== Workflow execution hangs on Windows ===<br />
<br />
This can happen if you are using "quick edit mode" in the console and have print statements in your code. Standard output can then get blocked by the console. Pressing space in the console resumes the execution. To avoid this problem, either disable "quick edit mode", or avoid print statements in your code.<br />
<br />
=== VisTrails do not install Missing System Packages ===<br />
<br />
If VisTrails do not try to install missing system packages it may be because it cannot determine your system type. I that case you can run this (in python) to determine your system type:<br />
<br />
import platform<br />
platform.linux_distribution()<br />
<br />
And add this system name to gui/bundles/utils.py by, e.g., modifying the _guess_ubuntu method (if your system is apt-based): <br />
<br />
def _guess_ubuntu():<br />
return platform.linux_distribution()[0]=='Ubuntu' or \<br />
platform.linux_distribution()[0]=='YourSystemName'<br />
<br />
=== Cannot update subworkflows after upgrading packages or vistrails version ===<br />
<br />
When packages used by a subworkflow is upgraded, any subworkflows that use it will be automatically upgraded. It may then lose the ability to be updated to a newer local subworkflow. In this case the subworkflow needs to be updated by hand by removing it from the pipeline and be dragged in again from the module palette. This may get fixed in a future release.<br />
<br />
== Building workflows ==<br />
<br />
=== Is there a way to give each widget a "display name" in addition to the module name at the center of the widget? ===<br />
<br />
Yes, a "display name" can be assigned to a module by selecting the triangle in its top right corner to open a popup menu and selecting the Set Module Label... menu item. You will then be prompted to enter the "display name". [http://www.vistrails.org/usersguide/v2.1/html/creating.html#configuring-module-labels Changing Module Labels]<br />
<br />
=== Is there a way to re-center the picture-in-picture (PiP) view? ===<br />
<br />
Yes. If you click on the PIP window to bring it to focus, you can press Ctrl-R (or Command-R on Mac) to re-center the PiP window. [http://www.vistrails.org/usersguide/v2.1/html/getting_started.html#vistrails-interaction Vistrails Interaction]<br />
<br />
=== How do I search for a literal "?" (question mark) in the search box in the Property panel? ===<br />
<br />
Since we allow regular expressions in our search box, question marks are treated as meta-characters. Thus, searching for "?" returns everything and "abc?" will return everything containing "abc". You need to use "\?" instead to search for "?". So the search for "??" would be "\?\?". [http://www.vistrails.org/usersguide/v2.1/html/querying.html#textual-queries Textual Queries]<br />
<br />
=== Saving a vistrail fails when Running VisTrails on Windows inside a Virtual Machine ===<br />
<br />
After installing Windows in a Virtual Machine, the path to zip.exe may be missing, and you may see this error when trying to save a vistrail:<br />
<br />
WindowsError: [Error 2] The system cannot find the file specified: '***/vt.zip'<br />
<br />
Then you need to add the path to zip.exe, which is included in the binary distribution of VisTrails, to your PATH variable.<br />
<br />
== Using VisTrails as a server ==<br />
<br />
=== What is the VisTrails server-mode? ===<br />
<br />
Using the VisTrails server mode, it is possible to ''execute workflows and control VisTrails through another application''. For example, the CrowdLabs Web portal (http://www.crowdlabs.org) accesses a VisTrails sever to execute workflows, retrieve and display vistrail trees and workflows. [http://www.vistrails.org/usersguide/v2.1/html/batch.html#using-vistrails-as-a-server Using VisTrails as a Server]<br />
<br />
=== How do I execute workflows and control VisTrails through another application? ===<br />
<br />
The way you access the server is by doing XML-RPC calls. In the current VisTrails release, we include a set of PHP scripts that can talk to a VisTrails server instance. They are in "extensions/http" folder. The files are reasonably well documented. Also, it should be not difficult to create python scripts to access the server (just use xmlrpclib module).<br />
<br />
Note that the VisTrails server requires the provenance and workflows to be in a database. More detailed instructions on how to setup the server and the database<br />
are available here:<br />
<br />
http://www.crowdlabs.org/site_media/static/dev_docs/vistrails_server_setup.html<br />
<br />
http://www.crowdlabs.org/site_media/static/dev_docs/vistrails_database_setup.html<br />
<br />
If what you want is just to execute a series of workflows in batch mode, a simpler solution would be to ''use the VisTrails client in batch mode''. Chapter 12 of the user's guide contains detailed information and examples on that. [http://www.vistrails.org/usersguide/v2.1/html/batch.html#sec-cli-batch Running VisTrails in Batch Mode]<br />
<br />
=== VisTrails server executes a workflow but generates a blank image and generates the error message ''cannot get access to X server'' ===<br />
<br />
You will need to check if the display the server is trying to use is a valid display (by default it uses the display 0). On linux, the command ''w'' will list the logged users and the display associated with them (''FROM'' column).<br />
<br />
Note that the VisTrails server requires the machine to be running X.<br />
<br />
=== cannot get access to X server ===<br />
<br />
Running VisTrails in '''server or batch mode''' requires a connection to an X server. <br />
<br />
No additional setup is required if you run VisTrails on a terminal because you are already logged in to X. To make it work in other scenarios, you need to run the python command through Xvfb or make sure you can run cgi scripts that access the GUI.<br />
<br />
If you can run Xvfb, you can use the following script, where you need toconfigure the first four variables according to your system: http://www.vistrails.org/images/Run_vistrails_batch_xvfb.script.sh.txt<br />
<br />
(To run the script, rename the file and remove the ".txt")<br />
<br />
<br />
You should also modify yout cgi script to invoke the bash script instead of vistrails directly. The bash script will accept the virtual display, the vistrail file and workflow tag as input arguments.<br />
<br />
Another possibility is if your workflow does not require the GUI, you can use VisTrails as a regular python module and it will not require the GUI or X Server to run. This functionality is available in the nightly builds and will be included in VisTrails 2.0 beta to be released soon. There is an example of how to use this feature in our FAQ: http://www.vistrails.org/index.php/FAQ#Using_VisTrails_as_a_Python_module<br />
<br />
== Problems starting VisTrails ==<br />
<br />
=== Setup was unable to create the directory "N:\.vistrails" ===<br />
<br />
When VisTrails is installing, it tries to create the .vistrails folder in the users %HOMEPATH% directory. In some Windows installations, network accounts are set to a directory that a user does not have write access to. Consequently, the installation will fail. To get around this problem, you can use the "-S <directory>" flag when starting VisTrails. This option allows you to put the .vistrails directory wherever you wish. You could also write a short script that automatically invokes VisTrails with the "-S" flag pointing to a directory that makes sense to your network. If you are unable to install VisTrails, you can run the installer after setting a new home path from the command line like this:<br />
<br />
set HOMEPATH=\My\New\Home\<br />
set HOMEDRIVE=C:<br />
vistrails-setup-2.0.1-xxx.exe<br />
<br />
== Using VisTrails as a Python module ==<br />
<br />
=== Can I use VisTrails as a Python module without installing PyQt? ===<br />
<br />
Yes! We have improved the ability to use VisTrails from other software, and have eliminated most GUI (PyQt) dependencies in the core part of the code. Thus, you can now work with workflow versions and provenance information in a standard python shell. Note packages that directly rely on the GUI like the VisTrails Spreadsheet will still require PyQt to be installed.<br />
<br />
=== How do I open and execute workflows in a standard python shell? ===<br />
<br />
Here is a simple example that shows how you can open and execute a workflow from a Python script:<br />
<br />
>>> import vistrails as vt<br />
>>><br />
>>> vistrail = vt.load_vistrail('simplemath.vt')<br />
>>> vistrail.select_latest_version()<br />
>>> result = vistrail.execute(in_a=2, in_b=4)<br />
>>> result.output_port('out_plus')<br />
6.0<br />
<br />
== Control Flow ==<br />
<br />
=== Note: using map ===<br />
<br />
When using 'map', the module (or subworkflow) used as function port in the map module MUST be a function, i.e., it can only define 1 output port. [http://www.vistrails.org/usersguide/v2.1/html/controlflow.html#the-map-operator The Map Operator]<br />
<br />
== Spreadsheet ==<br />
<br />
Where ''pipeline'' is a version number or a tag.<br />
<br />
=== How can I save an image from the spreadsheet? ===<br />
<br />
While having the focus on a spreadsheet cell and select the camera on the toolbar to take a snapshot. The system will prompt you for the location and file name where it should be saved. The other icons can be used for saving multiple images that can be used for generating an animation on demand. A whole sheet can also be saved by selecting Export (either from the menu or from the toolbar). [http://www.vistrails.org/usersguide/v2.1/html/spreadsheet.html#saving-a-spreadsheet-image Saving a Spreadsheet Image]<br />
<br />
===Is it possible to save the complete state of the spreadsheet?===<br />
[http://www.vistrails.org/usersguide/v2.1/html/spreadsheet.html#saving-a-spreadsheet Saving a Spreadsheet]<br />
<br />
=== Can I view multiple sheets at the same time? ===<br />
<br />
Yes. Each sheet on the spreadsheet can be displayed as a dock widget separated from the main spreadsheet window by dragging its tab name out of the tab bar at the bottom of the spreadsheet. [http://www.vistrails.org/usersguide/v2.1/html/spreadsheet.html#multiple-spreadsheets Multiple Spreadsheets]<br />
<br />
=== Then, how can I put back a separated sheet? ===<br />
<br />
A sheet can be docked back to the main window by dragging it back to the tab bar or double-click on its title bar. [http://www.vistrails.org/usersguide/v2.1/html/spreadsheet.html#multiple-spreadsheets Multiple Spreadsheets]<br />
<br />
=== How can I order sheets on the spreadsheet? ===<br />
<br />
This can be done by dragging the sheet name on the bottom top bar and drop it to the right place. [http://www.vistrails.org/usersguide/v2.1/html/spreadsheet.html#multiple-spreadsheets Multiple Spreadsheets]<br />
<br />
=== Can I control where a cell will be placed on the spreadsheet window? ===<br />
<br />
By default, an unoccupied cell on the active sheet will be chosen to display the result. However, you can specify exactly in the pipeline where a spreadsheet cell will be placed by using CellLocation and SheetReference. CellLocation specifies the location (row and column) of a cell when connecting to a spreadsheet cell (VTKCell, ImageViewerCell, ...). Similarly, a SheetReference module (when connecting to a CellLocation) will specify which sheet the cell will be put on given its name, minimum row size and minimum column size. There is an example of this in examples/vtk.xml (select the version below Double Renderer). [http://www.vistrails.org/usersguide/v2.1/html/spreadsheet.html#sending-output-to-the-spreadsheet Sending Output to the Spreadsheet]<br />
<br />
=== How do I output results to the spreadsheet? ===<br />
<br />
By inspecting the VisTrails Spreadsheet package (in the list of packages, to the left of the pipeline builder), you<br />
can see there are built-in cells for different kinds of data, e.g., RichTextCell to display HTML and plain text. op<br />
You (the user) can also define new cell types to display application-specific data. For example, we have developed<br />
VtkCell, MplFigureCell, and OpenGLCell. It is possible to display pretty much anything on the Spreadsheet! [http://www.vistrails.org/usersguide/v2.1/html/spreadsheet.html#sending-output-to-the-spreadsheet Sending Output to the Spreadsheet]<br />
<br />
Examples of writing cell modules can be found in:<br />
RichTextCell: packages/spreadsheet/widgets/richtext/richtext.py<br />
VTK: packages/vtk/vtkcell.py<br />
<br />
Here is the summary of some requirements on a cell widget:<br />
<br />
(1) It must be a Qt widget. It should inherit from spreadsheet_cell.QCellWidget in the spreadsheet package. Although any Qt Widget would work, certain features such as animation will not be available (without rewriting it).<br />
<br />
(2) It must re-implement the updateContents() function to take a set of inputs (usually coming from input ports of a wrapper Module) and display on the cells. VisTrails uses this function to update/reuse cells on the spreadsheet when new data comes in.<br />
<br />
(3) It needs a wrapper VisTrails Module (inherited from basic_widgets.SpreadsheetCell of the spreadsheet package). Inside the compute() method of this module, it may call self.display(CellWidgetType, (inputs)) to trigger the display event on the spreadsheet. [http://www.vistrails.org/usersguide/v2.1/html/spreadsheet.html#advanced-cell-options Advanced Cell Options]<br />
<br />
=== How do I control the default number of cells in the spreadsheet? ===<br />
<br />
You can configure the rowCount and colCount using the preferences dialog. Just go to the Module Packages tab, select spreadsheet in the "Enabled packages" and press the Configure button. Then a list of all the configuration options for the spreadsheet will show up. [http://www.vistrails.org/usersguide/v2.1/html/spreadsheet.html#custom-layout-options Custom Layout Options]<br />
<br />
=== Is it possible to launch a web browser from the vistrails spreadsheet? We would like to output several urls from a parameter sweep and then have the option to click on each one to view the resulting page. I can view the page within the spreadsheet, but it is really too crowded.===<br />
<br />
Currently, there isn't a widget that provides exactly this functionality, but I can think of a few solutions that may work for you:<br />
<br />
(1) You can use parameter exploration to generate multiple sheets so you might have an exploration that opens each page in a new sheet. Use the third column/dimension in the exploration interface to have a parameter span sheets.<br />
<br />
(2) The spreadsheet is extensible so you can write a custom spreadsheet cell widget that has a button or label with the desired link (a QLabel with openExternalLinks set to True, for example).<br />
<br />
(3) You can tweak the existing RichTextCell be adding the line "self.browser.setOpenExternalLinks(True)" at line 63 of the source file "vistrails/packages/spreadsheet/widgets/richtext/richtext.py". Then, if your workflow creates a file with html markup text like "<a href="http://www.vistrails.org/">VisTrails</a>" connected to a RichTextCell, clicking on the rendered link in the cell will open it in a web browser. You need to add the aforementioned line to the source to let Qt know that you want the link opened externally; by default, it will just issue an event that isn't processed. [http://www.vistrails.org/usersguide/v2.1/html/spreadsheet.html#launching-a-web-browser Launching a Web Browser]<br />
<br />
== Integrating your software into VisTrails ==<br />
<br />
===How can I integrate my own program into VisTrails?===<br />
<br />
The easiest way is to create a package. Writing a package is often very simple; please refer to [http://www.vistrails.org/usersguide/v2.1/html/packages.html this section of the users' guide].<br />
<br />
You can also dynamically generate modules. For an example see:<br />
<br />
[http://www.vistrails.org/usersguide/v2.1/html/packages.html#generating-modules-dynamically Generating Modules Dynamically]<br />
<br />
In particular, see the new_module call which uses python's type() function to generate new classes dynamically.<br />
<br />
===How do I add a port that is not visible on the module (when it appears on the design canvas)?===<br />
<br />
This can be accomplished via the "optional" argument. This is the fourth argument of add_input_port (add_output_port) or can be specified as a kwarg. In your example, this would look like:<br />
<br />
reg.add_input_port(MyModule, "MyPort", (core.modules.basic_modules.String, 'MyPort Name'), True)<br />
<br />
or with kwargs<br />
<br />
reg.add_input_port(MyModule, "MyPort", (core.modules.basic_modules.String, 'MyPort Name'),\<br />
optional=True)<br />
<br />
or<br />
<br />
_input_ports = [('MyPort', '(core.modules.basic_modules.String)', {"optional": True})]<br />
[http://www.vistrails.org/usersguide/v2.1/html/packages.html#configuring-ports Configuring Ports]<br />
<br />
=== How do modules deal with multiple inputs in a same port? ===<br />
<br />
(And should that even be allowed?)<br />
<br />
For compatibility reasons, we do need to allow multiple connections to an input port. However, most package developers should never have to use this, and so we do our best to hide it. the default behavior for getting inputs from a port, then, is to always return a single input.<br />
<br />
If on your module you need multiple inputs connected to a single port, use the 'forceGetInputListFromPort' method. It will return a list of all the data items coming through the port. The spreadsheet package uses this feature, so look there for usage examples (vistrails/packages/spreadsheet/basic_widgets.py)<br />
[http://www.vistrails.org/usersguide/v2.1/html/packages.html#configuring-ports Configuring Ports]<br />
<br />
=== Are there mechanisms for attaching widgets to different modules/parameters? ===<br />
<br />
Right now, we have a mechanism for putting a specific widget for an input port. For example, if a port is SetColor(red, green, blue), we can put a color wheel widget there. Or we can also replace the SetFileName port with a File Widget. However, this is not per parameter (only per port). We are currently working on this problem.<br />
<br />
=== Can I organize my package so it appears hierarchical in the module palette? ===<br />
<br />
Yes. Use the namespace keyword argument when adding the module to the registry. For example,<br />
<br />
registry.add_module(MyModule, namespace='MyNamespace')<br />
[http://www.vistrails.org/usersguide/v2.1/html/packages.html#configuring-modules Configuring Modules - Hierarchy and Visibility]<br />
<br />
=== Can I nest namespaces? ===<br />
<br />
Yes. Use the '|' character to separate different the hierarchy. For example,<br />
<br />
registry.add_module(MyModule, namespace='ParentNamespace|ChildNamespace')<br />
<br />
[http://www.vistrails.org/usersguide/v2.1/html/packages.html#configuring-modules Configuring Modules - Hierarchy and Visibility]<br />
<br />
=== Are there shortcuts for registry initialization? ===<br />
<br />
Yes. If you define _modules as a list of classes in the __init__.py file of your package, VisTrails will attempt to load all classes specified as modules. You can provide add_module options as keyword arguments by specifying a tuple (class, kwargs) in the list. For example:<br />
<br />
_modules = [MyModule1, (MyModule2, {'namespace': 'MyNamespace'})]<br />
<br />
In addition, you need to identify the ports of your modules as a field in your class by defining _input_ports and _output_ports lists. Here, the items in each list must be tuples of the form (portName, portSignature, optional=False, sort_key=-1). For example:<br />
<br />
class MyModule(Module):<br />
def compute(self):<br />
pass<br />
<br />
_input_ports = [('firstInput', String), ('secondInput', Integer, True)]<br />
_output_ports = [('firstOutput', String), ('secondOutput', String)]<br />
<br />
[http://www.vistrails.org/usersguide/v2.1/html/packages.html#customizing-modules-and-ports Customizing Modules and Ports]<br />
<br />
=== Can I define ports to be of types that I do not import into my package? ===<br />
<br />
Yes. You can pass an identifier string as the portSignature instead. The port_signature string is defined by:<br />
<br />
<module_string> := <package_identifier>:[<namespace>|]<module_name>,<br />
<port_signature> := (<module_string>*)<br />
<br />
For example,<br />
<br />
registry.add_input_port(MyModule, 'myInputPort', '(edu.utah.sci.vistrails.basic:String)')<br />
<br />
or<br />
<br />
_input_ports = [('myInputPort', '(edu.utah.sci.vistrails.basic:String)')]<br />
<br />
[http://www.vistrails.org/usersguide/v2.1/html/packages.html#configuring-ports Configuring Ports - Port Types]<br />
<br />
=== What do I need to change in my package to make it reloadable (new in v1.4.2)? ===<br />
See [http://www.vistrails.org/usersguide/v2.1/html/packages.html#creating-reloadable-packages Creating Reloadable Packages] for an explanation.<br />
<br />
=== Can I add default values or labels for parameters? ===<br />
<br />
Yes. Versions 1.4 and greater support these features. See [http://www.vistrails.org/usersguide/v2.1/html/packages.html#configuring-ports Configuring Ports - Default Values and Labels] for more details.<br />
<br />
=== How can I access the default values for a parameter? ===<br />
<br />
The default values are stored in PortSpec.defaults for each port.<br />
<br />
=== I want to write a module to load HDF data whose output (e.g., data, string) varies according to the input I give it. Is is possible to do this in VisTrails, and if yes, how can I do that? Ideally, I would like to avoid having to change the connection of my output every time I change the input. ===<br />
<br />
There are a few ways to tackle this - each has it's own benefits and pitfalls. Firstly, module connections do respect class hierarchies as we're familiar with in object oriented languages. For instance, A module can output a Constant of which String, Float, Integer, etc are specifications. In this way, you can have a subclass of something like HDFData be passed out of the module and the connections will be established regardless of the sub-type. This is a bit dangerous though. Modules downstream of such a class may not really know how to operate on certain types derived from the super-class. Extreme care must be taken both when creating the modules as well as connecting them to prevent things like this from happening.<br />
<br />
A second method that I employ in several different packages is the idea of a container class. For instance, the NumSciPy package uses a relatively generic container "Numpy Array" to encapsulate the data. Of course, these encapsulating objects can store dictionaries that other modules can easily access and understand how to operate on. Although this method is slightly more work, the benefits of a stricter typing of ports is beneficial - particularly upon interfacing with other packages that may depend on strongly typed constants (for example).<br />
[http://www.vistrails.org/usersguide/v2.1/html/packages.html#configuring-ports Varying Output According to the Input]<br />
<br />
=== I need to determine, at run-time, whether or not a "child" module is attached to the output port of a "parent" module. (I do not specifically need to know which child; just if there is one). ===<br />
<br />
The outputPorts dictionary of the base Module stores this information. Thus, you should be able to check<br />
<br />
("myPortName" in self.outputPorts)<br />
<br />
on the parent module to check if there are any downstream connections from the port "myPortName". This might be used, for example, to only set results for output ports that will be used. ***Note***, however, that the caching algorithm assumes that all outputs are set so adding a new connection to a previously unconnected output port will not work as desired if that module is cached. For this reason, I would currently recommend making such a module not cacheable. Another possibility is overriding the update() method to check the outputPorts and set the upToDate flag if they are not equal. In a single, limited test, this seemed to work, but be warned that it is not fully tested. Here is an example:<br />
<br />
class TestModule(Module):<br />
_output_ports = [('a1', '(edu.utah.sci.vistrails.basic:String)'),<br />
('a2', '(edu.utah.sci.vistrails.basic:String)')]<br />
def __init__(self):<br />
Module.__init__(self)<br />
self._cached_output_ports = set()<br />
<br />
def update(self):<br />
if len(set(self.outputPorts) - self._cached_output_ports) > 0:<br />
self.upToDate = False<br />
Module.update(self)<br />
<br />
def compute(self):<br />
if "a1" in self.outputPorts:<br />
self.setResult("a1", "test")<br />
if "a2" in self.outputPorts:<br />
self.setResult("a2", "test2")<br />
self._cached_output_ports = set(self.outputPorts)<br />
<br />
[http://www.vistrails.org/usersguide/v2.1/html/packages.html#configuring-ports Configuring Ports - Determining Whether or Not a Module is Attached to an Output Port]<br />
<br />
=== How can I make a module not display in the modules list? ===<br />
<br />
You should set the abstract parameter to True when adding the module to the registry. Using the original syntax, this looks like:<br />
<br />
def initialize():<br />
reg = core.modules.module_registry.get_module_registry()<br />
reg.add_module(InvisibleModule, abstract=True)<br />
# ...<br />
<br />
With the _modules dictionary shortcut (for more details, see [http://www.vistrails.org/index.php/FAQ#Are_there_shortcuts_for_registry_initialization.3F the FAQ section on this]), you include it in a kwargs dict as part of a module tuple:<br />
<br />
_modules = [AnotherModule, (InvisibleModule, {'abstract': True})]<br />
<br />
There is also a 'hide_descriptor' parameter that prevents the module from appearing in the module palette without declaring it to be abstract.<br />
<br />
The technical difference between the two is that 'abstract' will not add the item to the module palette while 'hide_descriptor' does add the item but immediately hides it. If the module should never be instantiated in a workflow, declare it abstract. If you don't want users to be able to add the module to a pipeline, but you have code that may add it programmatically, declare it with hide_descriptor=True.<br />
<br />
[http://www.vistrails.org/usersguide/v2.1/html/packages.html#configuring-modules Configuring Modules - Hierarchy and Visibility]<br />
<br />
=== How do I document individual ports? ===<br />
<br />
To access port documentation, users can right-click on the port in the port list and choose the corresponding menu item. To provide this documentation, you should define the <code>provide_input_port_documentation</code> and/or the <code>provide_output_port_documentation</code> '''class''' methods. Note that these methods take the class and the port name as arguments. For example,<br />
<br />
class MyModule(Module):<br />
_input_ports = [('test', '(edu.utah.sci.vistrails.basic:String)'),<br />
('test2', '(edu.utah.sci.vistrails.basic:String)')]<br />
port_docs = {'test': 'Some documentation',<br />
'test2': 'More documentation'}<br />
@classmethod<br />
def provide_input_port_documentation(cls, port_name):<br />
return cls.port_docs[port_name]<br />
<br />
=== How do I access modules from other packages? ===<br />
<br />
Currently, it is best to access modules from the registry. First, make sure that any dependencies on another package are specified in <code>package_dependencies</code> method in <code>__init__.py</code>. To create a module from another package as an output, you can generate it from the registry. For example,<br />
<br />
from core.modules.module_registry import get_module_registry<br />
from core.modules.vistrails_module import Module<br />
<br />
class ReturnFigManager(Module):<br />
_output_ports = [('figManager', <br />
'(edu.utah.sci.vistrails.matplotlib:MplFigureManager)')]<br />
def compute(self):<br />
reg = get_module_registry()<br />
wrapper = \<br />
reg.get_descriptor_by_name("edu.utah.sci.vistrails.matplotlib", <br />
"MplFigureManager").module()<br />
wrapper.figManager = "blah"<br />
self.setResult('figManager', wrapper)<br />
<br />
You can also create subclass from classes obtained from the registry. For example,<br />
<br />
MplFigureManager = get_module_registry().get_descriptor_by_name(<br />
"edu.utah.sci.vistrails.matplotlib", <br />
"MplFigureManager").module<br />
class MplFigureManagerSubclass(MplFigureManager):<br />
pass<br />
<br />
<br />
=== How do I create a custom module configuration widget? ===<br />
<br />
See [[Module Configuration Example]] for a full example and notes about doing this.<br />
<br />
=== Can I make a PythonSource module cacheable? ===<br />
<br />
Yes. If you have a module that you are planning to re-use in a workflow, we recommend making a packaged module (which are by default cacheable). However, you can make a PythonSource (which are by default not cacheable) cacheable using the line<br />
<br />
self.is_cacheable = lambda *args, **kwargs: True<br />
<br />
in the source of the PythonSource module.<br />
<br />
== The Console ==<br />
<br />
=== Where should I go to find out what I can call from the console and how to import it? ===<br />
<br />
We have tried to make some methods more accessible in the console via an api. You can import the api via <code>from vistrails import api</code> in the console and see the available methods with <code>dir(api)</code>. To open a vistrail:<br />
<br />
from vistrails import api<br />
api.open_vistrail_from_file('/Applications/VisTrails/examples/terminator.vt')<br />
<br />
To execute a version of a workflow, you currently have to go through the controller:<br />
<br />
api.select_version('Histogram')<br />
api.get_current_controller().execute_current_workflow()<br />
<br />
Currently, only a subset of VisTrails functionality is directly available from the api. However, since VisTrails is written in python, you can dig down starting with the VistrailsApplication or controller object to expose most of our internal methods. If you have suggestions for calls to be added to the api, please let us know.<br />
<br />
One other feature that we're working on, but is still in progress is the ability to construct workflows via the console. For example:<br />
<br />
vtk = load_package('edu.utah.sci.vistrails.vtk')<br />
vtk.vtkDataSetReader() # adds a vtkDataSetReader module to the pipeline<br />
# click on the new module<br />
a = selected_modules()[0] # get the one currently selected module<br />
a.SetFile('/vistrails/examples/data/head120.vtk') # sets the SetFile parmaeter for the data set reader<br />
b = vtk.vtkContourFilter() # adds a vtkContourFilter module to the pipeline and saves to var b<br />
b.SetInputConnection0(a.GetOutputPort0()) # connects a's GetOutputPort0 port to b's SetInputConnection0<br />
[http://www.vistrails.org/usersguide/v2.1/html/batch.html#finding-methods-via-the-command-line Finding Methods Via the Command Line]<br />
<br />
== Persistence Package ==<br />
<br />
=== How do I use the output of one workflow as the input for another using the persistence package? ===<br />
<br />
You need to configure the persistence modules using the module's configuration dialog. After adding a <code>PersistentOutputFile</code> to the workflow, click on the triangle in the upper-right corner of the <code>PersistentOutputFile</code>, and select "Edit Configuration" from the menu that appears. In this dialog, select "Create New Reference" and give the reference a name (and any space-delimited tags). Upon running that workflow, the data will be written to the persistent store. In the second workflow where you wish to use that file, add a <code>PersistentInputFile</code> and go to its configuration dialog in the same manner as with the output file. In that dialog, select "Use Existing Reference" and select the data that you just added in the first workflow in the list of files below. Now, when you run that workflow, it will grab the data from the persistent store.<br />
<br />
Here is an example: [[Media:offscreen_persistent.vt | offscreen_persistent.vt]]. Run the "persistent offscreen" workflow first, and then run the "display persistent output" to use the output of the first workflow as the input for the second.<br />
<br />
== VTK ==<br />
<br />
=== Given a VTK visualization, how can I generate a webpage from it? ===<br />
<br />
Check out the html pipeline in offscreen.xml.<br />
<br />
=== I'm trying to use VTK, but there doesn't seem to be any output. What is wrong? ===<br />
<br />
To use VTK on VisTrails, you need a slightly different way of connecting the renderer modules. Instead of using the standard RenderWindow/RenderWindowInteractor infrastructure, you simply connect the renderer to a VTKCell. The examples directory in the distribution has several VTK examples that illustrate.<br />
<br />
=== I am trying to add a module to the workflow via Python, but how can I access vtk modules? ===<br />
<br />
Here's an example:<br />
<br />
import api<br />
<br />
vtvtk = 'edu.utah.sci.vistrails.vtk'<br />
<br />
module = api.add_module(0, 0, vtvtk, 'vtkContourFilter', '')<br />
<br />
<br />
The third argument in add_module is the package identifier. You can find this in the "Module Packages" panel of the Preferences; just click on the package you're interested in and it will appear in the information on the right.<br />
<br />
== matplotlib ==<br />
<br />
=== I'm experiencing a problem with Latex labels and the matplotlib that comes with VisTrails 1.5. The script below entered to the interpreter that comes with VT is sufficient to reproduce it. ===<br />
<br />
import matplotlib.pyplot as plt<br />
plt.plot([1,2,3],[1,2,3])<br />
plt.xlabel("$foo$")<br />
<br />
Remove your ~/.matplotlib folder and re-start VisTrails<br />
<br />
== rpy ==<br />
<br />
=== Package rpy fails with "module object has no attribute RVector" ===<br />
<br />
The rpy package needs to be updated to support a newer rpy version. In "packages/rpy/init.py", replace all instances of "objects.RVector" with "objects.Vector", or use [https://raw.github.com/VisTrails/VisTrails/d0711a8d5ba8992662f69ed7924c625a3658b6a7/vistrails/packages/rpy/init.py this] file. <br />
<br />
== JobSubmission ==<br />
<br />
The JobSubmission package depends on the stable version of BatchQ. Download https://github.com/troelsfr/BatchQ/archive/stable.zip, copy the "BatchQ-stable/batchq" directory to your local site-packages folder. Copying it to the "vistrails/packages/JobSubmission" folder should also work. See batchq/contrib/vistrails for examples.<br />
<br />
== VisTrails Development ==<br />
<br />
=== I would like to build VisTrails from source. Are there instructions on how to do this? ===<br />
<br />
Yes! Take a look at [http://www.vistrails.org/usersguide/v2.1/html/getting_started.html#installing-vistrails-from-source Installing VisTrails from source]<br />
<br />
== Accessing Provenance Information ==<br />
<br />
=== How do I access the information in the execution log? ===<br />
<br />
The code responsible for storing execution information is located in the "core/log" directories, and the code that generates much of that information is in "core/interpreter/cached.py". Modules can add execution-specific annotations to provenance via annotate() calls during execution, but much of the data (like timing and errors) is captured by the LogController and CachedInterpreter (the execution engine) objects. To analyze the log from a vistrail (.vt) file, you might have something like the following:<br />
<br />
<code><br />
import core.log.log<br />
import db.services.io<br />
<br />
def run(fname):<br />
# open the .vt bundle specified by the filename "fname"<br />
bundle = db.services.io.open_vistrail_bundle_from_zip_xml(fname)[0]<br />
# get the log filename<br />
log_fname = bundle.vistrail.db_log_filename<br />
if log_fname is not None:<br />
# open the log<br />
log = db.services.io.open_log_from_xml(log_fname, True)<br />
# convert the log from a db object<br />
core.log.log.Log.convert(log)<br />
for workflow_exec in log.workflow_execs:<br />
print 'workflow version:', workflow_exec.parent_version<br />
print 'time started:', workflow_exec.ts_start<br />
print 'time ended:', workflow_exec.ts_end<br />
print 'modules executed:', [i.module_id <br />
for i in workflow_exec.item_execs]<br />
if __name__ == '__main__':<br />
run("some_vistrail.vt")<br />
</code><br />
<br />
You should be able to see what information is available by looking at the "core/log" classes. [http://www.vistrails.org/usersguide/v2.1/html/log.html Accessing the Execution Log]<br />
<br />
== VisTrails Binaries ==<br />
=== Is there a Mac OS X 10.6+ x64 binary of the version 1.7 of VisTrails available? ===<br />
<br />
We don't have a 64bit Mac binary for v1.7 release because at the time we didn't have 64 bit versions of the libraries shipped in the 1.7 binary.<br />
<br />
However, it is possible to update a 64bit or any other binary with a source release of VisTrails, including the sources of 1.7 version or the nightly builds.<br />
<br />
Assuming you have the sources of 1.7 in /vistrails1.7 and the 64bit binary in /Applications/VisTrails1.7 do the following steps:<br />
cp /vistrails1.7/vistrails/vistrails.py /Applications/VisTrails1.7/VisTrails.app/Contents/Resources<br />
cp -r /vistrails1.7/vistrails/api /Applications/VisTrails1.7/VisTrails.app/Contents/Resources/lib/python2.7/<br />
cp -r /vistrails1.7/vistrails/core /Applications/VisTrails1.7/VisTrails.app/Contents/Resources/lib/python2.7/<br />
cp -r /vistrails1.7/vistrails/db /Applications/VisTrails1.7/VisTrails.app/Contents/Resources/lib/python2.7/<br />
cp -r /vistrails1.7/vistrails/gui /Applications/VisTrails1.7/VisTrails.app/Contents/Resources/lib/python2.7/<br />
cp -r /vistrails1.7/vistrails/packages /Applications/VisTrails1.7/VisTrails.app/Contents/Resources/lib/python2.7/<br />
cp -r /vistrails1.7/examples /Applications/VisTrails1.7/<br />
cp -r /vistrails1.7/extensions /Applications/VisTrails1.7/<br />
cp -r /vistrails1.7/scripts /Applications/VisTrails1.7/<br />
<br />
=== AVG Antivirus falsely report a virus in VisTrails 2.0.2 32-bit Windows installer ===<br />
<br />
Problematic file is vistrails/Python27/Lib/site-packages/_mysql.pyd:<br />
https://www.virustotal.com/en/file/d8aabd921b5eba8aabcce936ce3b92e3d1de43eb44c43d921ca1b9ab91d7fd81/analysis/1366640335/. This is most likely a false positive and can be ignored.<br />
<br />
=== VisTrails fails after upgrading to OSX 10.9 ===<br />
<br />
Reinstalling [http://xquartz.macosforge.org/ XQuartz] should solve the problem.</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=13600Development2017-03-29T15:16:50Z<p>Remi: /* April 5, 2017 */</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2017 ==<br />
=== April 5, 2017 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
=== March 8, 2017 ===<br />
<br />
'''Updates'''<br />
* [RR] Added a looping indicator on modules [https://github.com/VisTrails/VisTrails/pull/1217 #1217]<br />
** This makes it easier to understand what's looping, no need to follow edges and count<br />
* [RR] Refactoring CLTools for integration in Docker package<br />
* [RR] Will try to refactor autogenerated package logic next (need to update TensorFlow package for TF 1.0, should really be using this)<br />
<br />
'''Items to Discuss'''<br />
* [RR] Problem with vtkRendererToSpreadsheet output mode: does it's own ModuleConnector handling, interfering with looping/streaming logic, to add the module ID on the renderers [https://github.com/VisTrails/VisTrails/issues/1216 #1216]<br />
** Can we move this up to the renderer modules?<br />
** Thankfully RR couldn't find much of this madness in the rest of code (outside of controlflow and parallelflow)<br />
<br />
=== February 24, 2017 ===<br />
<br />
'''Updates'''<br />
* [RR] Rich-text module documentation ([https://github.com/VisTrails/VisTrails/pull/1210 #1210])<br />
** Went over packages and fixed format; maybe we should have a test to validate RST?<br />
* [RR] Single-instance shutdown with <tt>--remote-shutdown</tt> ([https://github.com/VisTrails/VisTrails/pull/1204 #1204])<br />
* [RR] Bundle installation not super reliable; probably need to go in and add checks for every usual environment<br />
<br />
'''Items to Discuss'''<br />
* [RR] There are issues with list-depth<br />
** Some VTK ports marked with depth=1 don't actually accept multiple inputs (vtkDataSetMapper)<br />
** Issue with looping groups<br />
<br />
=== January 25, 2017 ===<br />
<br />
'''Updates'''<br />
<br />
<br />
'''Items to Discuss'''<br />
<br />
* Scaling VisTrails<br />
* sql-alchemy, uuid, bundle branches/merges<br />
<br />
=== January 11, 2017 ===<br />
<br />
'''Updates'''<br />
* [RR] New VisTrails version in Debian [https://packages.qa.debian.org/v/vistrails.html]<br />
<br />
'''Items to Discuss'''<br />
* [RR] Collaboration with school of medicine to port HiC-bench to VisTrails<br />
** Have homemade workflow system based on R scripts and symlinks<br />
*** Each steps has multiple sets of parameters<br />
*** Submits PBS jobs for each step with each set of parameters, using PBS job metadata to wait on previous step<br />
*** Gets all results in directory structure <code>step3-parameters/step2-parameters/step1-parameters/files</code><br />
** VisTrails improves provenance capture and makes editing pipelines easier<br />
*** Need better support for aggregation (groupby operation might be good enough; pandas?)<br />
<br />
== 2016 ==<br />
=== December 14, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Single-instance code: over-engineered, but will work<br />
** (Have to avoid race conditions on startup, and OSes offer very different locking primitives)<br />
<br />
'''Items to Discuss'''<br />
* Email from Matt Dirks: combobox should work, ask for more code?<br />
* Email from marin.nl:<br />
** looks like they would benefit from new interpreter<br />
** how to store their consolidated provenance when they exchange .vt files around?<br />
<br />
=== November 30, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Working on single instance code for MARIN ([https://github.com/VisTrails/VisTrails/pull/1205 #1205])<br />
** Remote shutdown ready ([https://github.com/VisTrails/VisTrails/pull/1204 #1204])<br />
** Still improving it<br />
** Want to return something more structured to the client (success/fail code, stdout, stderr)<br />
** Should be usable without Qt<br />
* [DK] Shutting down VisTrails has issues: cleans up interpreter (to remove files) but also resets persistent pipeline for no reason<br />
* [DK] Working on batch execution using console_mode<br />
* [DK] Settings aliased parameters when using core.api?<br />
<br />
=== November 16, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [RR] Issue [https://github.com/VisTrails/VisTrails/issues/1200 #1200] from @samtux: need to set PyQt to API version 2 before ever importing it; using require_pyqt_api2() does that (also can install bundle)<br />
<br />
'''Items to Discuss'''<br />
<br />
* Email from Pieter: executing pipelines with parameters from command-line using single-instance<br />
** Seems to work, main instance gets full command-line and alters configuration temporarily<br />
** Not sure what his error is (VisTrails returns little information in a lot of cases); though execution error should return something more meaningful<br />
** Actions:<br />
*** Can add switch to kill main instance<br />
*** Can improve single-instance code to return more meaningful errors + machine-readable output from "client" VisTrails<br />
*** Add client/server code to <tt>core</tt> that doesn't use Qt? Also need a run.py-line entrypoint that doesn't use gui (calls console mode like <tt>gui.application.VistrailsApplicationSingleton#noninteractiveMode()</tt>)?<br />
<br />
=== November 2, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* New version 2.2-2 of Debian package, still [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=796994 wrong copyright], and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=748692 broken bundle installation]<br />
** Finally replied, says copyright issue will be fixed soon, working on including VisTrails 2.2.4 for Debian Stretch<br />
* Working on Docker package, low hanging fruit because we have CLTools, and useful for some workflow (also step towards CWL support)<br />
<br />
'''Items to Discuss'''<br />
<br />
=== October 5, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
<br />
* working toward a 3.0 release; [https://github.com/VisTrails/VisTrails/projects/1 project board]<br />
** unique ids<br />
** bundles<br />
** GUI fixes<br />
** wrapping updates<br />
** python3<br />
** interpreter?<br />
<br />
* command-line input parameters to provenance<br />
** [DK] we don't create a persisted workflow from those input params, i think<br />
** [DK] these are already defined as aliases so we treat them differently form a normal input param<br />
** [DK] still have the potential problem of allowing a user to view the workflow as it was run with a given set of aliases<br />
** Do the vistrails variables values get stored in provenance? (what if values change?)<br />
<br />
=== September 14, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] [http://aosabook.org/en/vistrails.html VisTrails] in [http://aosabook.org/ The Architecture of Open Source Applications] Vol I (2012)<br />
** Did we know/contribute to this?<br />
*** [DK] Yes. We wrote the text for the VisTrails chapter<br />
** Drafted response<br />
*** [DK] Looks good, will forward.<br />
<br />
=== August 31, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Went over vgc.poly.edu/vistrails.org config:<br />
** Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)<br />
** Fixed usersguide scripts (won't disappear again)<br />
** Moved some websites over to HTTPS<br />
** [https://docs.google.com/document/d/18YZewjR2YY9cQ9pzdA-nU9jbu2L7jYcPlDi3iaPadwA/edit Websites]<br />
* [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs ([http://vistrails.readthedocs.io/ currently there]) and users' guide under the same RTD "project"<br />
* [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that<br />
<br />
'''Items to Discuss'''<br />
* [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/<br />
** Get rid of more services? (svn, trac, ...)<br />
* [RR] Notebook execution works. How do we support more languages (than Python)?<br />
** Need: detection, metadata reader, code to input values<br />
* [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?<br />
<br />
=== August 17, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Fixed [https://github.com/VisTrails/VisTrails/commit/2cd42d156243067a7663bb0bf0156e5d87188c2f parameter order upgrade bug] that breaks VTK examples.<br />
* [TE] Matplotlib and sklearn are exportable using module-to-script api<br />
* [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.<br />
<br />
'''Items to Discuss'''<br />
* Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!<br />
* Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)<br />
* [TE] Wrap-up notes<br />
<br />
=== August 10, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Python Wrapper and scripting<br />
** All changes merged into python-wrapper branch<br />
*** Tests pass<br />
*** Still need to update sub-branches<br />
** Minor fixes<br />
*** vtkPlot3DReader upgrade bug<br />
*** Added tuple exporting<br />
** Most of vtk examples can be exported [https://gist.github.com/rexissimus/5ac7374cde8525d84f9443f019d234e2 example]<br />
*** Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary<br />
**** Could be a bug in the old vistrail, but not sure<br />
* [DK] Subworkflows and uuids<br />
<br />
=== August 3, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Wrapper patches now stored as [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/vtk_patch.py python code]<br />
** Connection variables now lowercase<br />
** Can we use code as output variables?<br />
*** E.g. "vtk_dataset_mapper.GetOutput(0)"<br />
*** Could mutate output when output method is only supposed to be called once<br />
*** Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)<br />
<br />
=== July 27, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] DB bundle serialization implemented<br />
<br />
'''Items to Discuss'''<br />
* [TE] Finished using patch in script [https://github.com/VisTrails/VisTrails/tree/python-wrapper-script-patches python-wrapper-script-patches]<br />
** locale?<br />
** [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d new patching example] vs [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 old patching example]<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/parse.py#L135-L273 Example of new patches] <br />
** Legacy '.vtkInstance' is now added as a patch "self.vtkInstance=self" (see new patching example)<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/api.py New api for executing standalone modules]<br />
*** Accessing module was disabled for core/api, but a different api for this is probably better anyway.<br />
*** api.Package now gives [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d#file-new_patching-py-L26 directly callable module compute functions]<br />
*** Only works on modules not using interpreter functionality ('.interpreter', etc.)<br />
**** for zip file modules and others that need temp files and use the interpreter for this, need to modify the module, but could work around with python libraries for managing temp files<br />
** Fixed more renaming issues - [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/scripts.py#L63-L64 function argument names should not be replaced]<br />
** Still need to do something nicer for the patch formatting<br />
*** patch keys currently contain non-python-variable characters but this can be fixed<br />
*** Need to replace string template with python variables that is replacable using RedBaron<br />
**** Patches could then look like:<br />
<br />
def vtkInstance():<br />
self.vtkInstance = self<br />
<br />
=== July 20, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] New patch method for python-wrapper [https://gist.github.com/rexissimus/009b303e4cadcd4b8f2fec6cf848cb2b example]<br />
** A patch is a string template with self/input/output variables for patching a class method <br />
** Patches are stored in xml specification<br />
** Patches are assigned to methods in ClassSpec's<br />
** Used both for execution in vistrails and when exporting as script<br />
*** Executed with "exec" when run in vistrails<br />
*** Turns patches into natural calls in script<br />
** Wrapping VTK library no longer needed<br />
*** Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)<br />
** Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)<br />
** Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)<br />
** TODO's<br />
*** Add to export_as_script<br />
*** Patch Modules that represent Python functions as well<br />
<br />
* [vistrails-users] Read data from SQL example?<br />
<br />
=== July 13, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] bundles on sql-alchemy branch<br />
** Bundle, BundleMapping, and BundleSerializer<br />
** New BaseSerializers to deal with versions of directory, zip, db serializers<br />
** Can register the BundleSerializers with the base serializers<br />
** Translation between bundle versions<br />
** Bundle.add_object "just works" if an appropriate mapping was registered<br />
<br />
'''Items to Discuss'''<br />
* [TE] Simplified running module using API for scripting [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 example]<br />
** API Module can be executed using '''compute''' method<br />
*** Takes list of outputs to compute + inputs as keyword arguments<br />
* Status of general cell synchronization (vistrails-users)<br />
<br />
=== July 6, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] Working on bundles<br />
<br />
'''Items to Discuss'''<br />
* [TE] Using vistrails API in scripting [https://gist.github.com/rexissimus/034d9c4dedc3299a079e84981dedd8c7 example 1 (API with VTK)] [https://gist.github.com/rexissimus/a4a9ca6c08d4058c64967d2b35415fd5 example 2 (DownloadFile only)]<br />
** Call module in script using vistrails API<br />
** Can be used when no export method is defined.<br />
* [TE] [https://github.com/VisTrails/VisTrails/commit/c76c109349769fee1280f85c22678380a2f9cd39 Server news bug] Critical?<br />
**[RR] No, unreleased<br />
* [DK] New bundle organization<br />
** BundleMapping contains individual mappings from raw objects to BundleObj<br />
** Must be separate in order to have in-memory, separate from serializer settings<br />
<br />
=== June 29, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as python script<br />
** Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) [https://gist.github.com/rexissimus/16caf5200802e150296f62a0c7844c5a example]<br />
** Preludes (imports) still need improvements<br />
** Works with vtkInteractionHandler (clipping plane)<br />
** Resulting clipping plane is incorrect? (supposed to hide one side of the plane)<br />
** vtkRendererOutput Uses VTKRenderWindow as output<br />
*** Should support different output modes?<br />
*** Does not work with multiple outputs (Next cell is displayed when current is closed)<br />
*** Uses wrapped vtkInstance (as vvtk)<br />
**** Create new patch system with code that can be used both in vistrails and in a python script?<br />
<br />
=== June 22, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fix for upgraded analogies (PR [https://github.com/VisTrails/VisTrails/pull/1177 #1177])<br />
** Uses module location to find module remap (The best we can do right now?)<br />
** Also fixes remapped port names for connections to existing modules <br />
** May still fail on complex upgrades like adding a function on an upgraded module that has a different port name<br />
<br />
* [TE] Working on exporting wrapped libraries (VTK)<br />
<br />
=== June 15, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Added depth logic to python script exporter ([https://github.com/VisTrails/VisTrails/commit/50f220e4f6abe5a7bb8689f73f456f43df2dbc12 full commit])<br />
** Fully implements connection merging and list looping<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Making analogy work on upgraded workflows ([https://github.com/VisTrails/VisTrails/issues/1175 #1175])<br />
** Analogy only works We need to use the non-upgraded action chain<br />
** But pipelines need to be valid (matcher uses module portspecs)<br />
** Tried to get upgrade remap from upgrade action, but this seems impossible<br />
** Use module location instead?<br />
*** Location should be identical for one-to-one module upgrades<br />
<br />
=== June 1, 2016 ===<br />
<br />
'''Updates'''<br />
* VisTrails 2.2.4 announced (finally)<br />
** 5 survey replies<br />
** 47 downloads last week<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as script (python-wrapper)<br />
** to_python_script<br />
*** needs the pipeline (or at least the connections)<br />
**** Should not generate code for any possible VTK method (exporter detects which ports is used in the source)<br />
**** VTK needs to know connection order<br />
**** Adding functions directly to the script results in nicer code<br />
** Add support for multiple input connections<br />
** Add support for list looping<br />
<br />
=== May 25, 2016 ===<br />
<br />
'''Updates'''<br />
* Mailing-list fixed<br />
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])<br />
<br />
'''Items to Discuss'''<br />
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)<br />
* [TE] Union ports<br />
** Show union ports in pipeline (DEMO)<br />
<br />
=== May 18, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] mailing list still down - Julio working on it<br />
* [TE] Improved union port<br />
** New schema 1.0.5 adds portitem.union string attribute<br />
** Works with pipeline view, parameter exploration view, and mashup view<br />
** Should work well for matplotlib<br />
** Use with VTK ports?<br />
*** It uses [item_1,item_2] style ports usually with different tuple length.<br />
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]<br />
<br />
=== May 11, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?<br />
*** Shows single port for all ports with same sort_key<br />
*** Select specific type when adding/showing<br />
*** Only one type should be visible on the module at any time<br />
*** Making tuple parameters union types is probably too complicated<br />
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly) <br />
<br />
* [TE] Feature suggestion: Let connections connect to functions/parameters<br />
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters<br />
** Use separate views for ports and functions (like before)<br />
*** Allow rearranging of functions<br />
** Pros:<br />
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.<br />
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.<br />
**** Makes List module simpler<br />
*** Connect to parameter in tuple<br />
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds. <br />
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.<br />
** Cons:<br />
*** Interface more complicated and crowded<br />
*** More difficult for users to grasp?<br />
*** No immediate use case<br />
*** Makes no union type for tuple parameters a bigger problem.<br />
<br />
* [TE] Publish user survey?<br />
** No response from vistrails release yet<br />
** Not announced on vistrails-users yet.<br />
<br />
=== May 4, 2016 ===<br />
'''Updates'''<br />
* VisTrails 2.2.4 released<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
<br />
=== April 27, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Wrapping Bokeh<br />
** New method type "operation"<br />
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output<br />
**** Glyph sets function name and arguments as output<br />
***** Not serializable, but should always be computed together with the subject of the operation.<br />
**** Figure applies Glyph operation on itself<br />
*** Useful for matplotlib?<br />
**** matplotlib currently find actors by searching upstream?<br />
**** But this is more difficult since matplotlib is not function-based.<br />
** New "union" port type<br />
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value<br />
**** Uses only first value in tuple<br />
**** Bokeh have many properties that support (list, string, Float)<br />
*** Does not work with depth=1 types (e.g. Float of depth 1)<br />
**** Turn Depth 1 types with known type into lists?<br />
<br />
* [TE] New bugfix release<br />
** Finalize survey<br />
<br />
=== April 20, 2016 ===<br />
'''Updates'''<br />
* Merged upgrade fixes<br />
** Now has infinite loop test<br />
** loop test now correctly loops 5 times<br />
*** remap_module used wrong (current) package version when creating ModuleDescriptor<br />
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])<br />
<br />
'''Items to Discuss'''<br />
* New bugfix release<br />
** Merge usagestats<br />
** Need to finalize survey<br />
<br />
=== April 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?<br />
<br />
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])<br />
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]<br />
** Cherry-pick test to preserve source?<br />
** Need maxPipelineFixAttempts option?<br />
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.<br />
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?<br />
*** Would need to be moved into handle_invalid_pipeline<br />
<br />
=== April 6, 2016 ===<br />
'''Updates'''<br />
* [TE] Query View improvements<br />
** Done except for displaying "Version Info" and #1164<br />
* [RR] Working on finishing new interpreter (finally!)<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Try latest upgrade first, then previous.<br />
** If all fail, select latest upgrade?<br />
<br />
=== March 30, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Query View improvements<br />
** Unified do_version_switch and get_upgrades as validate_version()<br />
** Problems with query view using its own controller<br />
*** Fixed module info not being displayed for query results<br />
*** Still no version info for query results<br />
*** Still an issue with version query view not being updated<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== Older meetings ==<br />
* [[Development/2015]]<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=13599Development2017-03-29T14:50:11Z<p>Remi: /* March 29, 2017 */</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2017 ==<br />
=== March 29, 2017 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
=== March 8, 2017 ===<br />
<br />
'''Updates'''<br />
* [RR] Added a looping indicator on modules [https://github.com/VisTrails/VisTrails/pull/1217 #1217]<br />
** This makes it easier to understand what's looping, no need to follow edges and count<br />
* [RR] Refactoring CLTools for integration in Docker package<br />
* [RR] Will try to refactor autogenerated package logic next (need to update TensorFlow package for TF 1.0, should really be using this)<br />
<br />
'''Items to Discuss'''<br />
* [RR] Problem with vtkRendererToSpreadsheet output mode: does it's own ModuleConnector handling, interfering with looping/streaming logic, to add the module ID on the renderers [https://github.com/VisTrails/VisTrails/issues/1216 #1216]<br />
** Can we move this up to the renderer modules?<br />
** Thankfully RR couldn't find much of this madness in the rest of code (outside of controlflow and parallelflow)<br />
<br />
=== February 24, 2017 ===<br />
<br />
'''Updates'''<br />
* [RR] Rich-text module documentation ([https://github.com/VisTrails/VisTrails/pull/1210 #1210])<br />
** Went over packages and fixed format; maybe we should have a test to validate RST?<br />
* [RR] Single-instance shutdown with <tt>--remote-shutdown</tt> ([https://github.com/VisTrails/VisTrails/pull/1204 #1204])<br />
* [RR] Bundle installation not super reliable; probably need to go in and add checks for every usual environment<br />
<br />
'''Items to Discuss'''<br />
* [RR] There are issues with list-depth<br />
** Some VTK ports marked with depth=1 don't actually accept multiple inputs (vtkDataSetMapper)<br />
** Issue with looping groups<br />
<br />
=== January 25, 2017 ===<br />
<br />
'''Updates'''<br />
<br />
<br />
'''Items to Discuss'''<br />
<br />
* Scaling VisTrails<br />
* sql-alchemy, uuid, bundle branches/merges<br />
<br />
=== January 11, 2017 ===<br />
<br />
'''Updates'''<br />
* [RR] New VisTrails version in Debian [https://packages.qa.debian.org/v/vistrails.html]<br />
<br />
'''Items to Discuss'''<br />
* [RR] Collaboration with school of medicine to port HiC-bench to VisTrails<br />
** Have homemade workflow system based on R scripts and symlinks<br />
*** Each steps has multiple sets of parameters<br />
*** Submits PBS jobs for each step with each set of parameters, using PBS job metadata to wait on previous step<br />
*** Gets all results in directory structure <code>step3-parameters/step2-parameters/step1-parameters/files</code><br />
** VisTrails improves provenance capture and makes editing pipelines easier<br />
*** Need better support for aggregation (groupby operation might be good enough; pandas?)<br />
<br />
== 2016 ==<br />
=== December 14, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Single-instance code: over-engineered, but will work<br />
** (Have to avoid race conditions on startup, and OSes offer very different locking primitives)<br />
<br />
'''Items to Discuss'''<br />
* Email from Matt Dirks: combobox should work, ask for more code?<br />
* Email from marin.nl:<br />
** looks like they would benefit from new interpreter<br />
** how to store their consolidated provenance when they exchange .vt files around?<br />
<br />
=== November 30, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Working on single instance code for MARIN ([https://github.com/VisTrails/VisTrails/pull/1205 #1205])<br />
** Remote shutdown ready ([https://github.com/VisTrails/VisTrails/pull/1204 #1204])<br />
** Still improving it<br />
** Want to return something more structured to the client (success/fail code, stdout, stderr)<br />
** Should be usable without Qt<br />
* [DK] Shutting down VisTrails has issues: cleans up interpreter (to remove files) but also resets persistent pipeline for no reason<br />
* [DK] Working on batch execution using console_mode<br />
* [DK] Settings aliased parameters when using core.api?<br />
<br />
=== November 16, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [RR] Issue [https://github.com/VisTrails/VisTrails/issues/1200 #1200] from @samtux: need to set PyQt to API version 2 before ever importing it; using require_pyqt_api2() does that (also can install bundle)<br />
<br />
'''Items to Discuss'''<br />
<br />
* Email from Pieter: executing pipelines with parameters from command-line using single-instance<br />
** Seems to work, main instance gets full command-line and alters configuration temporarily<br />
** Not sure what his error is (VisTrails returns little information in a lot of cases); though execution error should return something more meaningful<br />
** Actions:<br />
*** Can add switch to kill main instance<br />
*** Can improve single-instance code to return more meaningful errors + machine-readable output from "client" VisTrails<br />
*** Add client/server code to <tt>core</tt> that doesn't use Qt? Also need a run.py-line entrypoint that doesn't use gui (calls console mode like <tt>gui.application.VistrailsApplicationSingleton#noninteractiveMode()</tt>)?<br />
<br />
=== November 2, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* New version 2.2-2 of Debian package, still [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=796994 wrong copyright], and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=748692 broken bundle installation]<br />
** Finally replied, says copyright issue will be fixed soon, working on including VisTrails 2.2.4 for Debian Stretch<br />
* Working on Docker package, low hanging fruit because we have CLTools, and useful for some workflow (also step towards CWL support)<br />
<br />
'''Items to Discuss'''<br />
<br />
=== October 5, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
<br />
* working toward a 3.0 release; [https://github.com/VisTrails/VisTrails/projects/1 project board]<br />
** unique ids<br />
** bundles<br />
** GUI fixes<br />
** wrapping updates<br />
** python3<br />
** interpreter?<br />
<br />
* command-line input parameters to provenance<br />
** [DK] we don't create a persisted workflow from those input params, i think<br />
** [DK] these are already defined as aliases so we treat them differently form a normal input param<br />
** [DK] still have the potential problem of allowing a user to view the workflow as it was run with a given set of aliases<br />
** Do the vistrails variables values get stored in provenance? (what if values change?)<br />
<br />
=== September 14, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] [http://aosabook.org/en/vistrails.html VisTrails] in [http://aosabook.org/ The Architecture of Open Source Applications] Vol I (2012)<br />
** Did we know/contribute to this?<br />
*** [DK] Yes. We wrote the text for the VisTrails chapter<br />
** Drafted response<br />
*** [DK] Looks good, will forward.<br />
<br />
=== August 31, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Went over vgc.poly.edu/vistrails.org config:<br />
** Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)<br />
** Fixed usersguide scripts (won't disappear again)<br />
** Moved some websites over to HTTPS<br />
** [https://docs.google.com/document/d/18YZewjR2YY9cQ9pzdA-nU9jbu2L7jYcPlDi3iaPadwA/edit Websites]<br />
* [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs ([http://vistrails.readthedocs.io/ currently there]) and users' guide under the same RTD "project"<br />
* [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that<br />
<br />
'''Items to Discuss'''<br />
* [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/<br />
** Get rid of more services? (svn, trac, ...)<br />
* [RR] Notebook execution works. How do we support more languages (than Python)?<br />
** Need: detection, metadata reader, code to input values<br />
* [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?<br />
<br />
=== August 17, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Fixed [https://github.com/VisTrails/VisTrails/commit/2cd42d156243067a7663bb0bf0156e5d87188c2f parameter order upgrade bug] that breaks VTK examples.<br />
* [TE] Matplotlib and sklearn are exportable using module-to-script api<br />
* [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.<br />
<br />
'''Items to Discuss'''<br />
* Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!<br />
* Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)<br />
* [TE] Wrap-up notes<br />
<br />
=== August 10, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Python Wrapper and scripting<br />
** All changes merged into python-wrapper branch<br />
*** Tests pass<br />
*** Still need to update sub-branches<br />
** Minor fixes<br />
*** vtkPlot3DReader upgrade bug<br />
*** Added tuple exporting<br />
** Most of vtk examples can be exported [https://gist.github.com/rexissimus/5ac7374cde8525d84f9443f019d234e2 example]<br />
*** Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary<br />
**** Could be a bug in the old vistrail, but not sure<br />
* [DK] Subworkflows and uuids<br />
<br />
=== August 3, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Wrapper patches now stored as [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/vtk_patch.py python code]<br />
** Connection variables now lowercase<br />
** Can we use code as output variables?<br />
*** E.g. "vtk_dataset_mapper.GetOutput(0)"<br />
*** Could mutate output when output method is only supposed to be called once<br />
*** Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)<br />
<br />
=== July 27, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] DB bundle serialization implemented<br />
<br />
'''Items to Discuss'''<br />
* [TE] Finished using patch in script [https://github.com/VisTrails/VisTrails/tree/python-wrapper-script-patches python-wrapper-script-patches]<br />
** locale?<br />
** [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d new patching example] vs [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 old patching example]<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/parse.py#L135-L273 Example of new patches] <br />
** Legacy '.vtkInstance' is now added as a patch "self.vtkInstance=self" (see new patching example)<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/api.py New api for executing standalone modules]<br />
*** Accessing module was disabled for core/api, but a different api for this is probably better anyway.<br />
*** api.Package now gives [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d#file-new_patching-py-L26 directly callable module compute functions]<br />
*** Only works on modules not using interpreter functionality ('.interpreter', etc.)<br />
**** for zip file modules and others that need temp files and use the interpreter for this, need to modify the module, but could work around with python libraries for managing temp files<br />
** Fixed more renaming issues - [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/scripts.py#L63-L64 function argument names should not be replaced]<br />
** Still need to do something nicer for the patch formatting<br />
*** patch keys currently contain non-python-variable characters but this can be fixed<br />
*** Need to replace string template with python variables that is replacable using RedBaron<br />
**** Patches could then look like:<br />
<br />
def vtkInstance():<br />
self.vtkInstance = self<br />
<br />
=== July 20, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] New patch method for python-wrapper [https://gist.github.com/rexissimus/009b303e4cadcd4b8f2fec6cf848cb2b example]<br />
** A patch is a string template with self/input/output variables for patching a class method <br />
** Patches are stored in xml specification<br />
** Patches are assigned to methods in ClassSpec's<br />
** Used both for execution in vistrails and when exporting as script<br />
*** Executed with "exec" when run in vistrails<br />
*** Turns patches into natural calls in script<br />
** Wrapping VTK library no longer needed<br />
*** Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)<br />
** Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)<br />
** Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)<br />
** TODO's<br />
*** Add to export_as_script<br />
*** Patch Modules that represent Python functions as well<br />
<br />
* [vistrails-users] Read data from SQL example?<br />
<br />
=== July 13, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] bundles on sql-alchemy branch<br />
** Bundle, BundleMapping, and BundleSerializer<br />
** New BaseSerializers to deal with versions of directory, zip, db serializers<br />
** Can register the BundleSerializers with the base serializers<br />
** Translation between bundle versions<br />
** Bundle.add_object "just works" if an appropriate mapping was registered<br />
<br />
'''Items to Discuss'''<br />
* [TE] Simplified running module using API for scripting [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 example]<br />
** API Module can be executed using '''compute''' method<br />
*** Takes list of outputs to compute + inputs as keyword arguments<br />
* Status of general cell synchronization (vistrails-users)<br />
<br />
=== July 6, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] Working on bundles<br />
<br />
'''Items to Discuss'''<br />
* [TE] Using vistrails API in scripting [https://gist.github.com/rexissimus/034d9c4dedc3299a079e84981dedd8c7 example 1 (API with VTK)] [https://gist.github.com/rexissimus/a4a9ca6c08d4058c64967d2b35415fd5 example 2 (DownloadFile only)]<br />
** Call module in script using vistrails API<br />
** Can be used when no export method is defined.<br />
* [TE] [https://github.com/VisTrails/VisTrails/commit/c76c109349769fee1280f85c22678380a2f9cd39 Server news bug] Critical?<br />
**[RR] No, unreleased<br />
* [DK] New bundle organization<br />
** BundleMapping contains individual mappings from raw objects to BundleObj<br />
** Must be separate in order to have in-memory, separate from serializer settings<br />
<br />
=== June 29, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as python script<br />
** Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) [https://gist.github.com/rexissimus/16caf5200802e150296f62a0c7844c5a example]<br />
** Preludes (imports) still need improvements<br />
** Works with vtkInteractionHandler (clipping plane)<br />
** Resulting clipping plane is incorrect? (supposed to hide one side of the plane)<br />
** vtkRendererOutput Uses VTKRenderWindow as output<br />
*** Should support different output modes?<br />
*** Does not work with multiple outputs (Next cell is displayed when current is closed)<br />
*** Uses wrapped vtkInstance (as vvtk)<br />
**** Create new patch system with code that can be used both in vistrails and in a python script?<br />
<br />
=== June 22, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fix for upgraded analogies (PR [https://github.com/VisTrails/VisTrails/pull/1177 #1177])<br />
** Uses module location to find module remap (The best we can do right now?)<br />
** Also fixes remapped port names for connections to existing modules <br />
** May still fail on complex upgrades like adding a function on an upgraded module that has a different port name<br />
<br />
* [TE] Working on exporting wrapped libraries (VTK)<br />
<br />
=== June 15, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Added depth logic to python script exporter ([https://github.com/VisTrails/VisTrails/commit/50f220e4f6abe5a7bb8689f73f456f43df2dbc12 full commit])<br />
** Fully implements connection merging and list looping<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Making analogy work on upgraded workflows ([https://github.com/VisTrails/VisTrails/issues/1175 #1175])<br />
** Analogy only works We need to use the non-upgraded action chain<br />
** But pipelines need to be valid (matcher uses module portspecs)<br />
** Tried to get upgrade remap from upgrade action, but this seems impossible<br />
** Use module location instead?<br />
*** Location should be identical for one-to-one module upgrades<br />
<br />
=== June 1, 2016 ===<br />
<br />
'''Updates'''<br />
* VisTrails 2.2.4 announced (finally)<br />
** 5 survey replies<br />
** 47 downloads last week<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as script (python-wrapper)<br />
** to_python_script<br />
*** needs the pipeline (or at least the connections)<br />
**** Should not generate code for any possible VTK method (exporter detects which ports is used in the source)<br />
**** VTK needs to know connection order<br />
**** Adding functions directly to the script results in nicer code<br />
** Add support for multiple input connections<br />
** Add support for list looping<br />
<br />
=== May 25, 2016 ===<br />
<br />
'''Updates'''<br />
* Mailing-list fixed<br />
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])<br />
<br />
'''Items to Discuss'''<br />
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)<br />
* [TE] Union ports<br />
** Show union ports in pipeline (DEMO)<br />
<br />
=== May 18, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] mailing list still down - Julio working on it<br />
* [TE] Improved union port<br />
** New schema 1.0.5 adds portitem.union string attribute<br />
** Works with pipeline view, parameter exploration view, and mashup view<br />
** Should work well for matplotlib<br />
** Use with VTK ports?<br />
*** It uses [item_1,item_2] style ports usually with different tuple length.<br />
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]<br />
<br />
=== May 11, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?<br />
*** Shows single port for all ports with same sort_key<br />
*** Select specific type when adding/showing<br />
*** Only one type should be visible on the module at any time<br />
*** Making tuple parameters union types is probably too complicated<br />
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly) <br />
<br />
* [TE] Feature suggestion: Let connections connect to functions/parameters<br />
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters<br />
** Use separate views for ports and functions (like before)<br />
*** Allow rearranging of functions<br />
** Pros:<br />
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.<br />
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.<br />
**** Makes List module simpler<br />
*** Connect to parameter in tuple<br />
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds. <br />
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.<br />
** Cons:<br />
*** Interface more complicated and crowded<br />
*** More difficult for users to grasp?<br />
*** No immediate use case<br />
*** Makes no union type for tuple parameters a bigger problem.<br />
<br />
* [TE] Publish user survey?<br />
** No response from vistrails release yet<br />
** Not announced on vistrails-users yet.<br />
<br />
=== May 4, 2016 ===<br />
'''Updates'''<br />
* VisTrails 2.2.4 released<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
<br />
=== April 27, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Wrapping Bokeh<br />
** New method type "operation"<br />
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output<br />
**** Glyph sets function name and arguments as output<br />
***** Not serializable, but should always be computed together with the subject of the operation.<br />
**** Figure applies Glyph operation on itself<br />
*** Useful for matplotlib?<br />
**** matplotlib currently find actors by searching upstream?<br />
**** But this is more difficult since matplotlib is not function-based.<br />
** New "union" port type<br />
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value<br />
**** Uses only first value in tuple<br />
**** Bokeh have many properties that support (list, string, Float)<br />
*** Does not work with depth=1 types (e.g. Float of depth 1)<br />
**** Turn Depth 1 types with known type into lists?<br />
<br />
* [TE] New bugfix release<br />
** Finalize survey<br />
<br />
=== April 20, 2016 ===<br />
'''Updates'''<br />
* Merged upgrade fixes<br />
** Now has infinite loop test<br />
** loop test now correctly loops 5 times<br />
*** remap_module used wrong (current) package version when creating ModuleDescriptor<br />
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])<br />
<br />
'''Items to Discuss'''<br />
* New bugfix release<br />
** Merge usagestats<br />
** Need to finalize survey<br />
<br />
=== April 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?<br />
<br />
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])<br />
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]<br />
** Cherry-pick test to preserve source?<br />
** Need maxPipelineFixAttempts option?<br />
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.<br />
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?<br />
*** Would need to be moved into handle_invalid_pipeline<br />
<br />
=== April 6, 2016 ===<br />
'''Updates'''<br />
* [TE] Query View improvements<br />
** Done except for displaying "Version Info" and #1164<br />
* [RR] Working on finishing new interpreter (finally!)<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Try latest upgrade first, then previous.<br />
** If all fail, select latest upgrade?<br />
<br />
=== March 30, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Query View improvements<br />
** Unified do_version_switch and get_upgrades as validate_version()<br />
** Problems with query view using its own controller<br />
*** Fixed module info not being displayed for query results<br />
*** Still no version info for query results<br />
*** Still an issue with version query view not being updated<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== Older meetings ==<br />
* [[Development/2015]]<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=13442Development2017-03-08T15:31:05Z<p>Remi: /* March 8, 2017 */</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2017 ==<br />
=== March 8, 2017 ===<br />
<br />
'''Updates'''<br />
* [RR] Added a looping indicator on modules [https://github.com/VisTrails/VisTrails/pull/1217 #1217]<br />
** This makes it easier to understand what's looping, no need to follow edges and count<br />
* [RR] Refactoring CLTools for integration in Docker package<br />
* [RR] Will try to refactor autogenerated package logic next (need to update TensorFlow package for TF 1.0, should really be using this)<br />
<br />
'''Items to Discuss'''<br />
* [RR] Problem with vtkRendererToSpreadsheet output mode: does it's own ModuleConnector handling, interfering with looping/streaming logic, to add the module ID on the renderers [https://github.com/VisTrails/VisTrails/issues/1216 #1216]<br />
** Can we move this up to the renderer modules?<br />
** Thankfully RR couldn't find much of this madness in the rest of code (outside of controlflow and parallelflow)<br />
<br />
=== February 24, 2017 ===<br />
<br />
'''Updates'''<br />
* [RR] Rich-text module documentation ([https://github.com/VisTrails/VisTrails/pull/1210 #1210])<br />
** Went over packages and fixed format; maybe we should have a test to validate RST?<br />
* [RR] Single-instance shutdown with <tt>--remote-shutdown</tt> ([https://github.com/VisTrails/VisTrails/pull/1204 #1204])<br />
* [RR] Bundle installation not super reliable; probably need to go in and add checks for every usual environment<br />
<br />
'''Items to Discuss'''<br />
* [RR] There are issues with list-depth<br />
** Some VTK ports marked with depth=1 don't actually accept multiple inputs (vtkDataSetMapper)<br />
** Issue with looping groups<br />
<br />
=== January 25, 2017 ===<br />
<br />
'''Updates'''<br />
<br />
<br />
'''Items to Discuss'''<br />
<br />
* Scaling VisTrails<br />
* sql-alchemy, uuid, bundle branches/merges<br />
<br />
=== January 11, 2017 ===<br />
<br />
'''Updates'''<br />
* [RR] New VisTrails version in Debian [https://packages.qa.debian.org/v/vistrails.html]<br />
<br />
'''Items to Discuss'''<br />
* [RR] Collaboration with school of medicine to port HiC-bench to VisTrails<br />
** Have homemade workflow system based on R scripts and symlinks<br />
*** Each steps has multiple sets of parameters<br />
*** Submits PBS jobs for each step with each set of parameters, using PBS job metadata to wait on previous step<br />
*** Gets all results in directory structure <code>step3-parameters/step2-parameters/step1-parameters/files</code><br />
** VisTrails improves provenance capture and makes editing pipelines easier<br />
*** Need better support for aggregation (groupby operation might be good enough; pandas?)<br />
<br />
== 2016 ==<br />
=== December 14, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Single-instance code: over-engineered, but will work<br />
** (Have to avoid race conditions on startup, and OSes offer very different locking primitives)<br />
<br />
'''Items to Discuss'''<br />
* Email from Matt Dirks: combobox should work, ask for more code?<br />
* Email from marin.nl:<br />
** looks like they would benefit from new interpreter<br />
** how to store their consolidated provenance when they exchange .vt files around?<br />
<br />
=== November 30, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Working on single instance code for MARIN ([https://github.com/VisTrails/VisTrails/pull/1205 #1205])<br />
** Remote shutdown ready ([https://github.com/VisTrails/VisTrails/pull/1204 #1204])<br />
** Still improving it<br />
** Want to return something more structured to the client (success/fail code, stdout, stderr)<br />
** Should be usable without Qt<br />
* [DK] Shutting down VisTrails has issues: cleans up interpreter (to remove files) but also resets persistent pipeline for no reason<br />
* [DK] Working on batch execution using console_mode<br />
* [DK] Settings aliased parameters when using core.api?<br />
<br />
=== November 16, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [RR] Issue [https://github.com/VisTrails/VisTrails/issues/1200 #1200] from @samtux: need to set PyQt to API version 2 before ever importing it; using require_pyqt_api2() does that (also can install bundle)<br />
<br />
'''Items to Discuss'''<br />
<br />
* Email from Pieter: executing pipelines with parameters from command-line using single-instance<br />
** Seems to work, main instance gets full command-line and alters configuration temporarily<br />
** Not sure what his error is (VisTrails returns little information in a lot of cases); though execution error should return something more meaningful<br />
** Actions:<br />
*** Can add switch to kill main instance<br />
*** Can improve single-instance code to return more meaningful errors + machine-readable output from "client" VisTrails<br />
*** Add client/server code to <tt>core</tt> that doesn't use Qt? Also need a run.py-line entrypoint that doesn't use gui (calls console mode like <tt>gui.application.VistrailsApplicationSingleton#noninteractiveMode()</tt>)?<br />
<br />
=== November 2, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* New version 2.2-2 of Debian package, still [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=796994 wrong copyright], and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=748692 broken bundle installation]<br />
** Finally replied, says copyright issue will be fixed soon, working on including VisTrails 2.2.4 for Debian Stretch<br />
* Working on Docker package, low hanging fruit because we have CLTools, and useful for some workflow (also step towards CWL support)<br />
<br />
'''Items to Discuss'''<br />
<br />
=== October 5, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
<br />
* working toward a 3.0 release; [https://github.com/VisTrails/VisTrails/projects/1 project board]<br />
** unique ids<br />
** bundles<br />
** GUI fixes<br />
** wrapping updates<br />
** python3<br />
** interpreter?<br />
<br />
* command-line input parameters to provenance<br />
** [DK] we don't create a persisted workflow from those input params, i think<br />
** [DK] these are already defined as aliases so we treat them differently form a normal input param<br />
** [DK] still have the potential problem of allowing a user to view the workflow as it was run with a given set of aliases<br />
** Do the vistrails variables values get stored in provenance? (what if values change?)<br />
<br />
=== September 14, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] [http://aosabook.org/en/vistrails.html VisTrails] in [http://aosabook.org/ The Architecture of Open Source Applications] Vol I (2012)<br />
** Did we know/contribute to this?<br />
*** [DK] Yes. We wrote the text for the VisTrails chapter<br />
** Drafted response<br />
*** [DK] Looks good, will forward.<br />
<br />
=== August 31, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Went over vgc.poly.edu/vistrails.org config:<br />
** Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)<br />
** Fixed usersguide scripts (won't disappear again)<br />
** Moved some websites over to HTTPS<br />
** [https://docs.google.com/document/d/18YZewjR2YY9cQ9pzdA-nU9jbu2L7jYcPlDi3iaPadwA/edit Websites]<br />
* [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs ([http://vistrails.readthedocs.io/ currently there]) and users' guide under the same RTD "project"<br />
* [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that<br />
<br />
'''Items to Discuss'''<br />
* [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/<br />
** Get rid of more services? (svn, trac, ...)<br />
* [RR] Notebook execution works. How do we support more languages (than Python)?<br />
** Need: detection, metadata reader, code to input values<br />
* [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?<br />
<br />
=== August 17, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Fixed [https://github.com/VisTrails/VisTrails/commit/2cd42d156243067a7663bb0bf0156e5d87188c2f parameter order upgrade bug] that breaks VTK examples.<br />
* [TE] Matplotlib and sklearn are exportable using module-to-script api<br />
* [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.<br />
<br />
'''Items to Discuss'''<br />
* Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!<br />
* Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)<br />
* [TE] Wrap-up notes<br />
<br />
=== August 10, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Python Wrapper and scripting<br />
** All changes merged into python-wrapper branch<br />
*** Tests pass<br />
*** Still need to update sub-branches<br />
** Minor fixes<br />
*** vtkPlot3DReader upgrade bug<br />
*** Added tuple exporting<br />
** Most of vtk examples can be exported [https://gist.github.com/rexissimus/5ac7374cde8525d84f9443f019d234e2 example]<br />
*** Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary<br />
**** Could be a bug in the old vistrail, but not sure<br />
* [DK] Subworkflows and uuids<br />
<br />
=== August 3, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Wrapper patches now stored as [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/vtk_patch.py python code]<br />
** Connection variables now lowercase<br />
** Can we use code as output variables?<br />
*** E.g. "vtk_dataset_mapper.GetOutput(0)"<br />
*** Could mutate output when output method is only supposed to be called once<br />
*** Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)<br />
<br />
=== July 27, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] DB bundle serialization implemented<br />
<br />
'''Items to Discuss'''<br />
* [TE] Finished using patch in script [https://github.com/VisTrails/VisTrails/tree/python-wrapper-script-patches python-wrapper-script-patches]<br />
** locale?<br />
** [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d new patching example] vs [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 old patching example]<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/parse.py#L135-L273 Example of new patches] <br />
** Legacy '.vtkInstance' is now added as a patch "self.vtkInstance=self" (see new patching example)<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/api.py New api for executing standalone modules]<br />
*** Accessing module was disabled for core/api, but a different api for this is probably better anyway.<br />
*** api.Package now gives [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d#file-new_patching-py-L26 directly callable module compute functions]<br />
*** Only works on modules not using interpreter functionality ('.interpreter', etc.)<br />
**** for zip file modules and others that need temp files and use the interpreter for this, need to modify the module, but could work around with python libraries for managing temp files<br />
** Fixed more renaming issues - [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/scripts.py#L63-L64 function argument names should not be replaced]<br />
** Still need to do something nicer for the patch formatting<br />
*** patch keys currently contain non-python-variable characters but this can be fixed<br />
*** Need to replace string template with python variables that is replacable using RedBaron<br />
**** Patches could then look like:<br />
<br />
def vtkInstance():<br />
self.vtkInstance = self<br />
<br />
=== July 20, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] New patch method for python-wrapper [https://gist.github.com/rexissimus/009b303e4cadcd4b8f2fec6cf848cb2b example]<br />
** A patch is a string template with self/input/output variables for patching a class method <br />
** Patches are stored in xml specification<br />
** Patches are assigned to methods in ClassSpec's<br />
** Used both for execution in vistrails and when exporting as script<br />
*** Executed with "exec" when run in vistrails<br />
*** Turns patches into natural calls in script<br />
** Wrapping VTK library no longer needed<br />
*** Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)<br />
** Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)<br />
** Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)<br />
** TODO's<br />
*** Add to export_as_script<br />
*** Patch Modules that represent Python functions as well<br />
<br />
* [vistrails-users] Read data from SQL example?<br />
<br />
=== July 13, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] bundles on sql-alchemy branch<br />
** Bundle, BundleMapping, and BundleSerializer<br />
** New BaseSerializers to deal with versions of directory, zip, db serializers<br />
** Can register the BundleSerializers with the base serializers<br />
** Translation between bundle versions<br />
** Bundle.add_object "just works" if an appropriate mapping was registered<br />
<br />
'''Items to Discuss'''<br />
* [TE] Simplified running module using API for scripting [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 example]<br />
** API Module can be executed using '''compute''' method<br />
*** Takes list of outputs to compute + inputs as keyword arguments<br />
* Status of general cell synchronization (vistrails-users)<br />
<br />
=== July 6, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] Working on bundles<br />
<br />
'''Items to Discuss'''<br />
* [TE] Using vistrails API in scripting [https://gist.github.com/rexissimus/034d9c4dedc3299a079e84981dedd8c7 example 1 (API with VTK)] [https://gist.github.com/rexissimus/a4a9ca6c08d4058c64967d2b35415fd5 example 2 (DownloadFile only)]<br />
** Call module in script using vistrails API<br />
** Can be used when no export method is defined.<br />
* [TE] [https://github.com/VisTrails/VisTrails/commit/c76c109349769fee1280f85c22678380a2f9cd39 Server news bug] Critical?<br />
**[RR] No, unreleased<br />
* [DK] New bundle organization<br />
** BundleMapping contains individual mappings from raw objects to BundleObj<br />
** Must be separate in order to have in-memory, separate from serializer settings<br />
<br />
=== June 29, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as python script<br />
** Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) [https://gist.github.com/rexissimus/16caf5200802e150296f62a0c7844c5a example]<br />
** Preludes (imports) still need improvements<br />
** Works with vtkInteractionHandler (clipping plane)<br />
** Resulting clipping plane is incorrect? (supposed to hide one side of the plane)<br />
** vtkRendererOutput Uses VTKRenderWindow as output<br />
*** Should support different output modes?<br />
*** Does not work with multiple outputs (Next cell is displayed when current is closed)<br />
*** Uses wrapped vtkInstance (as vvtk)<br />
**** Create new patch system with code that can be used both in vistrails and in a python script?<br />
<br />
=== June 22, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fix for upgraded analogies (PR [https://github.com/VisTrails/VisTrails/pull/1177 #1177])<br />
** Uses module location to find module remap (The best we can do right now?)<br />
** Also fixes remapped port names for connections to existing modules <br />
** May still fail on complex upgrades like adding a function on an upgraded module that has a different port name<br />
<br />
* [TE] Working on exporting wrapped libraries (VTK)<br />
<br />
=== June 15, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Added depth logic to python script exporter ([https://github.com/VisTrails/VisTrails/commit/50f220e4f6abe5a7bb8689f73f456f43df2dbc12 full commit])<br />
** Fully implements connection merging and list looping<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Making analogy work on upgraded workflows ([https://github.com/VisTrails/VisTrails/issues/1175 #1175])<br />
** Analogy only works We need to use the non-upgraded action chain<br />
** But pipelines need to be valid (matcher uses module portspecs)<br />
** Tried to get upgrade remap from upgrade action, but this seems impossible<br />
** Use module location instead?<br />
*** Location should be identical for one-to-one module upgrades<br />
<br />
=== June 1, 2016 ===<br />
<br />
'''Updates'''<br />
* VisTrails 2.2.4 announced (finally)<br />
** 5 survey replies<br />
** 47 downloads last week<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as script (python-wrapper)<br />
** to_python_script<br />
*** needs the pipeline (or at least the connections)<br />
**** Should not generate code for any possible VTK method (exporter detects which ports is used in the source)<br />
**** VTK needs to know connection order<br />
**** Adding functions directly to the script results in nicer code<br />
** Add support for multiple input connections<br />
** Add support for list looping<br />
<br />
=== May 25, 2016 ===<br />
<br />
'''Updates'''<br />
* Mailing-list fixed<br />
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])<br />
<br />
'''Items to Discuss'''<br />
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)<br />
* [TE] Union ports<br />
** Show union ports in pipeline (DEMO)<br />
<br />
=== May 18, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] mailing list still down - Julio working on it<br />
* [TE] Improved union port<br />
** New schema 1.0.5 adds portitem.union string attribute<br />
** Works with pipeline view, parameter exploration view, and mashup view<br />
** Should work well for matplotlib<br />
** Use with VTK ports?<br />
*** It uses [item_1,item_2] style ports usually with different tuple length.<br />
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]<br />
<br />
=== May 11, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?<br />
*** Shows single port for all ports with same sort_key<br />
*** Select specific type when adding/showing<br />
*** Only one type should be visible on the module at any time<br />
*** Making tuple parameters union types is probably too complicated<br />
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly) <br />
<br />
* [TE] Feature suggestion: Let connections connect to functions/parameters<br />
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters<br />
** Use separate views for ports and functions (like before)<br />
*** Allow rearranging of functions<br />
** Pros:<br />
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.<br />
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.<br />
**** Makes List module simpler<br />
*** Connect to parameter in tuple<br />
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds. <br />
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.<br />
** Cons:<br />
*** Interface more complicated and crowded<br />
*** More difficult for users to grasp?<br />
*** No immediate use case<br />
*** Makes no union type for tuple parameters a bigger problem.<br />
<br />
* [TE] Publish user survey?<br />
** No response from vistrails release yet<br />
** Not announced on vistrails-users yet.<br />
<br />
=== May 4, 2016 ===<br />
'''Updates'''<br />
* VisTrails 2.2.4 released<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
<br />
=== April 27, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Wrapping Bokeh<br />
** New method type "operation"<br />
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output<br />
**** Glyph sets function name and arguments as output<br />
***** Not serializable, but should always be computed together with the subject of the operation.<br />
**** Figure applies Glyph operation on itself<br />
*** Useful for matplotlib?<br />
**** matplotlib currently find actors by searching upstream?<br />
**** But this is more difficult since matplotlib is not function-based.<br />
** New "union" port type<br />
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value<br />
**** Uses only first value in tuple<br />
**** Bokeh have many properties that support (list, string, Float)<br />
*** Does not work with depth=1 types (e.g. Float of depth 1)<br />
**** Turn Depth 1 types with known type into lists?<br />
<br />
* [TE] New bugfix release<br />
** Finalize survey<br />
<br />
=== April 20, 2016 ===<br />
'''Updates'''<br />
* Merged upgrade fixes<br />
** Now has infinite loop test<br />
** loop test now correctly loops 5 times<br />
*** remap_module used wrong (current) package version when creating ModuleDescriptor<br />
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])<br />
<br />
'''Items to Discuss'''<br />
* New bugfix release<br />
** Merge usagestats<br />
** Need to finalize survey<br />
<br />
=== April 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?<br />
<br />
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])<br />
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]<br />
** Cherry-pick test to preserve source?<br />
** Need maxPipelineFixAttempts option?<br />
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.<br />
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?<br />
*** Would need to be moved into handle_invalid_pipeline<br />
<br />
=== April 6, 2016 ===<br />
'''Updates'''<br />
* [TE] Query View improvements<br />
** Done except for displaying "Version Info" and #1164<br />
* [RR] Working on finishing new interpreter (finally!)<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Try latest upgrade first, then previous.<br />
** If all fail, select latest upgrade?<br />
<br />
=== March 30, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Query View improvements<br />
** Unified do_version_switch and get_upgrades as validate_version()<br />
** Problems with query view using its own controller<br />
*** Fixed module info not being displayed for query results<br />
*** Still no version info for query results<br />
*** Still an issue with version query view not being updated<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== Older meetings ==<br />
* [[Development/2015]]<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=13331Development2017-02-24T19:10:13Z<p>Remi: /* February 24, 2017 */</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2017 ==<br />
=== February 24, 2017 ===<br />
<br />
'''Updates'''<br />
* [RR] Rich-text module documentation ([https://github.com/VisTrails/VisTrails/pull/1210 #1210])<br />
** Went over packages and fixed format; maybe we should have a test to validate RST?<br />
* [RR] Single-instance shutdown with <tt>--remote-shutdown</tt> ([https://github.com/VisTrails/VisTrails/pull/1204 #1204])<br />
* [RR] Bundle installation not super reliable; probably need to go in and add checks for every usual environment<br />
<br />
'''Items to Discuss'''<br />
* [RR] There are issues with list-depth<br />
** Some VTK ports marked with depth=1 don't actually accept multiple inputs (vtkDataSetMapper)<br />
** Issue with looping groups<br />
<br />
=== January 25, 2017 ===<br />
<br />
'''Updates'''<br />
<br />
<br />
'''Items to Discuss'''<br />
<br />
* Scaling VisTrails<br />
* sql-alchemy, uuid, bundle branches/merges<br />
<br />
=== January 11, 2017 ===<br />
<br />
'''Updates'''<br />
* [RR] New VisTrails version in Debian [https://packages.qa.debian.org/v/vistrails.html]<br />
<br />
'''Items to Discuss'''<br />
* [RR] Collaboration with school of medicine to port HiC-bench to VisTrails<br />
** Have homemade workflow system based on R scripts and symlinks<br />
*** Each steps has multiple sets of parameters<br />
*** Submits PBS jobs for each step with each set of parameters, using PBS job metadata to wait on previous step<br />
*** Gets all results in directory structure <code>step3-parameters/step2-parameters/step1-parameters/files</code><br />
** VisTrails improves provenance capture and makes editing pipelines easier<br />
*** Need better support for aggregation (groupby operation might be good enough; pandas?)<br />
<br />
== 2016 ==<br />
=== December 14, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Single-instance code: over-engineered, but will work<br />
** (Have to avoid race conditions on startup, and OSes offer very different locking primitives)<br />
<br />
'''Items to Discuss'''<br />
* Email from Matt Dirks: combobox should work, ask for more code?<br />
* Email from marin.nl:<br />
** looks like they would benefit from new interpreter<br />
** how to store their consolidated provenance when they exchange .vt files around?<br />
<br />
=== November 30, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Working on single instance code for MARIN ([https://github.com/VisTrails/VisTrails/pull/1205 #1205])<br />
** Remote shutdown ready ([https://github.com/VisTrails/VisTrails/pull/1204 #1204])<br />
** Still improving it<br />
** Want to return something more structured to the client (success/fail code, stdout, stderr)<br />
** Should be usable without Qt<br />
* [DK] Shutting down VisTrails has issues: cleans up interpreter (to remove files) but also resets persistent pipeline for no reason<br />
* [DK] Working on batch execution using console_mode<br />
* [DK] Settings aliased parameters when using core.api?<br />
<br />
=== November 16, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [RR] Issue [https://github.com/VisTrails/VisTrails/issues/1200 #1200] from @samtux: need to set PyQt to API version 2 before ever importing it; using require_pyqt_api2() does that (also can install bundle)<br />
<br />
'''Items to Discuss'''<br />
<br />
* Email from Pieter: executing pipelines with parameters from command-line using single-instance<br />
** Seems to work, main instance gets full command-line and alters configuration temporarily<br />
** Not sure what his error is (VisTrails returns little information in a lot of cases); though execution error should return something more meaningful<br />
** Actions:<br />
*** Can add switch to kill main instance<br />
*** Can improve single-instance code to return more meaningful errors + machine-readable output from "client" VisTrails<br />
*** Add client/server code to <tt>core</tt> that doesn't use Qt? Also need a run.py-line entrypoint that doesn't use gui (calls console mode like <tt>gui.application.VistrailsApplicationSingleton#noninteractiveMode()</tt>)?<br />
<br />
=== November 2, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* New version 2.2-2 of Debian package, still [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=796994 wrong copyright], and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=748692 broken bundle installation]<br />
** Finally replied, says copyright issue will be fixed soon, working on including VisTrails 2.2.4 for Debian Stretch<br />
* Working on Docker package, low hanging fruit because we have CLTools, and useful for some workflow (also step towards CWL support)<br />
<br />
'''Items to Discuss'''<br />
<br />
=== October 5, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
<br />
* working toward a 3.0 release; [https://github.com/VisTrails/VisTrails/projects/1 project board]<br />
** unique ids<br />
** bundles<br />
** GUI fixes<br />
** wrapping updates<br />
** python3<br />
** interpreter?<br />
<br />
* command-line input parameters to provenance<br />
** [DK] we don't create a persisted workflow from those input params, i think<br />
** [DK] these are already defined as aliases so we treat them differently form a normal input param<br />
** [DK] still have the potential problem of allowing a user to view the workflow as it was run with a given set of aliases<br />
** Do the vistrails variables values get stored in provenance? (what if values change?)<br />
<br />
=== September 14, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] [http://aosabook.org/en/vistrails.html VisTrails] in [http://aosabook.org/ The Architecture of Open Source Applications] Vol I (2012)<br />
** Did we know/contribute to this?<br />
*** [DK] Yes. We wrote the text for the VisTrails chapter<br />
** Drafted response<br />
*** [DK] Looks good, will forward.<br />
<br />
=== August 31, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Went over vgc.poly.edu/vistrails.org config:<br />
** Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)<br />
** Fixed usersguide scripts (won't disappear again)<br />
** Moved some websites over to HTTPS<br />
** [https://docs.google.com/document/d/18YZewjR2YY9cQ9pzdA-nU9jbu2L7jYcPlDi3iaPadwA/edit Websites]<br />
* [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs ([http://vistrails.readthedocs.io/ currently there]) and users' guide under the same RTD "project"<br />
* [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that<br />
<br />
'''Items to Discuss'''<br />
* [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/<br />
** Get rid of more services? (svn, trac, ...)<br />
* [RR] Notebook execution works. How do we support more languages (than Python)?<br />
** Need: detection, metadata reader, code to input values<br />
* [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?<br />
<br />
=== August 17, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Fixed [https://github.com/VisTrails/VisTrails/commit/2cd42d156243067a7663bb0bf0156e5d87188c2f parameter order upgrade bug] that breaks VTK examples.<br />
* [TE] Matplotlib and sklearn are exportable using module-to-script api<br />
* [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.<br />
<br />
'''Items to Discuss'''<br />
* Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!<br />
* Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)<br />
* [TE] Wrap-up notes<br />
<br />
=== August 10, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Python Wrapper and scripting<br />
** All changes merged into python-wrapper branch<br />
*** Tests pass<br />
*** Still need to update sub-branches<br />
** Minor fixes<br />
*** vtkPlot3DReader upgrade bug<br />
*** Added tuple exporting<br />
** Most of vtk examples can be exported [https://gist.github.com/rexissimus/5ac7374cde8525d84f9443f019d234e2 example]<br />
*** Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary<br />
**** Could be a bug in the old vistrail, but not sure<br />
* [DK] Subworkflows and uuids<br />
<br />
=== August 3, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Wrapper patches now stored as [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/vtk_patch.py python code]<br />
** Connection variables now lowercase<br />
** Can we use code as output variables?<br />
*** E.g. "vtk_dataset_mapper.GetOutput(0)"<br />
*** Could mutate output when output method is only supposed to be called once<br />
*** Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)<br />
<br />
=== July 27, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] DB bundle serialization implemented<br />
<br />
'''Items to Discuss'''<br />
* [TE] Finished using patch in script [https://github.com/VisTrails/VisTrails/tree/python-wrapper-script-patches python-wrapper-script-patches]<br />
** locale?<br />
** [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d new patching example] vs [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 old patching example]<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/parse.py#L135-L273 Example of new patches] <br />
** Legacy '.vtkInstance' is now added as a patch "self.vtkInstance=self" (see new patching example)<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/api.py New api for executing standalone modules]<br />
*** Accessing module was disabled for core/api, but a different api for this is probably better anyway.<br />
*** api.Package now gives [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d#file-new_patching-py-L26 directly callable module compute functions]<br />
*** Only works on modules not using interpreter functionality ('.interpreter', etc.)<br />
**** for zip file modules and others that need temp files and use the interpreter for this, need to modify the module, but could work around with python libraries for managing temp files<br />
** Fixed more renaming issues - [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/scripts.py#L63-L64 function argument names should not be replaced]<br />
** Still need to do something nicer for the patch formatting<br />
*** patch keys currently contain non-python-variable characters but this can be fixed<br />
*** Need to replace string template with python variables that is replacable using RedBaron<br />
**** Patches could then look like:<br />
<br />
def vtkInstance():<br />
self.vtkInstance = self<br />
<br />
=== July 20, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] New patch method for python-wrapper [https://gist.github.com/rexissimus/009b303e4cadcd4b8f2fec6cf848cb2b example]<br />
** A patch is a string template with self/input/output variables for patching a class method <br />
** Patches are stored in xml specification<br />
** Patches are assigned to methods in ClassSpec's<br />
** Used both for execution in vistrails and when exporting as script<br />
*** Executed with "exec" when run in vistrails<br />
*** Turns patches into natural calls in script<br />
** Wrapping VTK library no longer needed<br />
*** Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)<br />
** Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)<br />
** Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)<br />
** TODO's<br />
*** Add to export_as_script<br />
*** Patch Modules that represent Python functions as well<br />
<br />
* [vistrails-users] Read data from SQL example?<br />
<br />
=== July 13, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] bundles on sql-alchemy branch<br />
** Bundle, BundleMapping, and BundleSerializer<br />
** New BaseSerializers to deal with versions of directory, zip, db serializers<br />
** Can register the BundleSerializers with the base serializers<br />
** Translation between bundle versions<br />
** Bundle.add_object "just works" if an appropriate mapping was registered<br />
<br />
'''Items to Discuss'''<br />
* [TE] Simplified running module using API for scripting [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 example]<br />
** API Module can be executed using '''compute''' method<br />
*** Takes list of outputs to compute + inputs as keyword arguments<br />
* Status of general cell synchronization (vistrails-users)<br />
<br />
=== July 6, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] Working on bundles<br />
<br />
'''Items to Discuss'''<br />
* [TE] Using vistrails API in scripting [https://gist.github.com/rexissimus/034d9c4dedc3299a079e84981dedd8c7 example 1 (API with VTK)] [https://gist.github.com/rexissimus/a4a9ca6c08d4058c64967d2b35415fd5 example 2 (DownloadFile only)]<br />
** Call module in script using vistrails API<br />
** Can be used when no export method is defined.<br />
* [TE] [https://github.com/VisTrails/VisTrails/commit/c76c109349769fee1280f85c22678380a2f9cd39 Server news bug] Critical?<br />
**[RR] No, unreleased<br />
* [DK] New bundle organization<br />
** BundleMapping contains individual mappings from raw objects to BundleObj<br />
** Must be separate in order to have in-memory, separate from serializer settings<br />
<br />
=== June 29, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as python script<br />
** Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) [https://gist.github.com/rexissimus/16caf5200802e150296f62a0c7844c5a example]<br />
** Preludes (imports) still need improvements<br />
** Works with vtkInteractionHandler (clipping plane)<br />
** Resulting clipping plane is incorrect? (supposed to hide one side of the plane)<br />
** vtkRendererOutput Uses VTKRenderWindow as output<br />
*** Should support different output modes?<br />
*** Does not work with multiple outputs (Next cell is displayed when current is closed)<br />
*** Uses wrapped vtkInstance (as vvtk)<br />
**** Create new patch system with code that can be used both in vistrails and in a python script?<br />
<br />
=== June 22, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fix for upgraded analogies (PR [https://github.com/VisTrails/VisTrails/pull/1177 #1177])<br />
** Uses module location to find module remap (The best we can do right now?)<br />
** Also fixes remapped port names for connections to existing modules <br />
** May still fail on complex upgrades like adding a function on an upgraded module that has a different port name<br />
<br />
* [TE] Working on exporting wrapped libraries (VTK)<br />
<br />
=== June 15, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Added depth logic to python script exporter ([https://github.com/VisTrails/VisTrails/commit/50f220e4f6abe5a7bb8689f73f456f43df2dbc12 full commit])<br />
** Fully implements connection merging and list looping<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Making analogy work on upgraded workflows ([https://github.com/VisTrails/VisTrails/issues/1175 #1175])<br />
** Analogy only works We need to use the non-upgraded action chain<br />
** But pipelines need to be valid (matcher uses module portspecs)<br />
** Tried to get upgrade remap from upgrade action, but this seems impossible<br />
** Use module location instead?<br />
*** Location should be identical for one-to-one module upgrades<br />
<br />
=== June 1, 2016 ===<br />
<br />
'''Updates'''<br />
* VisTrails 2.2.4 announced (finally)<br />
** 5 survey replies<br />
** 47 downloads last week<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as script (python-wrapper)<br />
** to_python_script<br />
*** needs the pipeline (or at least the connections)<br />
**** Should not generate code for any possible VTK method (exporter detects which ports is used in the source)<br />
**** VTK needs to know connection order<br />
**** Adding functions directly to the script results in nicer code<br />
** Add support for multiple input connections<br />
** Add support for list looping<br />
<br />
=== May 25, 2016 ===<br />
<br />
'''Updates'''<br />
* Mailing-list fixed<br />
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])<br />
<br />
'''Items to Discuss'''<br />
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)<br />
* [TE] Union ports<br />
** Show union ports in pipeline (DEMO)<br />
<br />
=== May 18, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] mailing list still down - Julio working on it<br />
* [TE] Improved union port<br />
** New schema 1.0.5 adds portitem.union string attribute<br />
** Works with pipeline view, parameter exploration view, and mashup view<br />
** Should work well for matplotlib<br />
** Use with VTK ports?<br />
*** It uses [item_1,item_2] style ports usually with different tuple length.<br />
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]<br />
<br />
=== May 11, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?<br />
*** Shows single port for all ports with same sort_key<br />
*** Select specific type when adding/showing<br />
*** Only one type should be visible on the module at any time<br />
*** Making tuple parameters union types is probably too complicated<br />
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly) <br />
<br />
* [TE] Feature suggestion: Let connections connect to functions/parameters<br />
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters<br />
** Use separate views for ports and functions (like before)<br />
*** Allow rearranging of functions<br />
** Pros:<br />
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.<br />
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.<br />
**** Makes List module simpler<br />
*** Connect to parameter in tuple<br />
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds. <br />
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.<br />
** Cons:<br />
*** Interface more complicated and crowded<br />
*** More difficult for users to grasp?<br />
*** No immediate use case<br />
*** Makes no union type for tuple parameters a bigger problem.<br />
<br />
* [TE] Publish user survey?<br />
** No response from vistrails release yet<br />
** Not announced on vistrails-users yet.<br />
<br />
=== May 4, 2016 ===<br />
'''Updates'''<br />
* VisTrails 2.2.4 released<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
<br />
=== April 27, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Wrapping Bokeh<br />
** New method type "operation"<br />
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output<br />
**** Glyph sets function name and arguments as output<br />
***** Not serializable, but should always be computed together with the subject of the operation.<br />
**** Figure applies Glyph operation on itself<br />
*** Useful for matplotlib?<br />
**** matplotlib currently find actors by searching upstream?<br />
**** But this is more difficult since matplotlib is not function-based.<br />
** New "union" port type<br />
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value<br />
**** Uses only first value in tuple<br />
**** Bokeh have many properties that support (list, string, Float)<br />
*** Does not work with depth=1 types (e.g. Float of depth 1)<br />
**** Turn Depth 1 types with known type into lists?<br />
<br />
* [TE] New bugfix release<br />
** Finalize survey<br />
<br />
=== April 20, 2016 ===<br />
'''Updates'''<br />
* Merged upgrade fixes<br />
** Now has infinite loop test<br />
** loop test now correctly loops 5 times<br />
*** remap_module used wrong (current) package version when creating ModuleDescriptor<br />
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])<br />
<br />
'''Items to Discuss'''<br />
* New bugfix release<br />
** Merge usagestats<br />
** Need to finalize survey<br />
<br />
=== April 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?<br />
<br />
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])<br />
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]<br />
** Cherry-pick test to preserve source?<br />
** Need maxPipelineFixAttempts option?<br />
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.<br />
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?<br />
*** Would need to be moved into handle_invalid_pipeline<br />
<br />
=== April 6, 2016 ===<br />
'''Updates'''<br />
* [TE] Query View improvements<br />
** Done except for displaying "Version Info" and #1164<br />
* [RR] Working on finishing new interpreter (finally!)<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Try latest upgrade first, then previous.<br />
** If all fail, select latest upgrade?<br />
<br />
=== March 30, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Query View improvements<br />
** Unified do_version_switch and get_upgrades as validate_version()<br />
** Problems with query view using its own controller<br />
*** Fixed module info not being displayed for query results<br />
*** Still no version info for query results<br />
*** Still an issue with version query view not being updated<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== Older meetings ==<br />
* [[Development/2015]]<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=13330Development2017-02-24T19:08:49Z<p>Remi: /* February 24, 2017 */</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2017 ==<br />
=== February 24, 2017 ===<br />
<br />
'''Updates'''<br />
* [RR] Rich-text module documentation ([https://github.com/VisTrails/VisTrails/pull/1210 #1210])<br />
** Went over packages and fixed format; maybe we should have a test to validate RST?<br />
* [RR] Single-instance shutdown with <tt>--remote-shutdown</tt> ([https://github.com/VisTrails/VisTrails/pull/1204 #1204])<br />
<br />
'''Items to Discuss'''<br />
* [RR] There are issues with list-depth<br />
** Some VTK ports marked with depth=1 don't actually accept multiple inputs (vtkDataSetMapper)<br />
** Issue with looping groups<br />
<br />
=== January 25, 2017 ===<br />
<br />
'''Updates'''<br />
<br />
<br />
'''Items to Discuss'''<br />
<br />
* Scaling VisTrails<br />
* sql-alchemy, uuid, bundle branches/merges<br />
<br />
=== January 11, 2017 ===<br />
<br />
'''Updates'''<br />
* [RR] New VisTrails version in Debian [https://packages.qa.debian.org/v/vistrails.html]<br />
<br />
'''Items to Discuss'''<br />
* [RR] Collaboration with school of medicine to port HiC-bench to VisTrails<br />
** Have homemade workflow system based on R scripts and symlinks<br />
*** Each steps has multiple sets of parameters<br />
*** Submits PBS jobs for each step with each set of parameters, using PBS job metadata to wait on previous step<br />
*** Gets all results in directory structure <code>step3-parameters/step2-parameters/step1-parameters/files</code><br />
** VisTrails improves provenance capture and makes editing pipelines easier<br />
*** Need better support for aggregation (groupby operation might be good enough; pandas?)<br />
<br />
== 2016 ==<br />
=== December 14, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Single-instance code: over-engineered, but will work<br />
** (Have to avoid race conditions on startup, and OSes offer very different locking primitives)<br />
<br />
'''Items to Discuss'''<br />
* Email from Matt Dirks: combobox should work, ask for more code?<br />
* Email from marin.nl:<br />
** looks like they would benefit from new interpreter<br />
** how to store their consolidated provenance when they exchange .vt files around?<br />
<br />
=== November 30, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Working on single instance code for MARIN ([https://github.com/VisTrails/VisTrails/pull/1205 #1205])<br />
** Remote shutdown ready ([https://github.com/VisTrails/VisTrails/pull/1204 #1204])<br />
** Still improving it<br />
** Want to return something more structured to the client (success/fail code, stdout, stderr)<br />
** Should be usable without Qt<br />
* [DK] Shutting down VisTrails has issues: cleans up interpreter (to remove files) but also resets persistent pipeline for no reason<br />
* [DK] Working on batch execution using console_mode<br />
* [DK] Settings aliased parameters when using core.api?<br />
<br />
=== November 16, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [RR] Issue [https://github.com/VisTrails/VisTrails/issues/1200 #1200] from @samtux: need to set PyQt to API version 2 before ever importing it; using require_pyqt_api2() does that (also can install bundle)<br />
<br />
'''Items to Discuss'''<br />
<br />
* Email from Pieter: executing pipelines with parameters from command-line using single-instance<br />
** Seems to work, main instance gets full command-line and alters configuration temporarily<br />
** Not sure what his error is (VisTrails returns little information in a lot of cases); though execution error should return something more meaningful<br />
** Actions:<br />
*** Can add switch to kill main instance<br />
*** Can improve single-instance code to return more meaningful errors + machine-readable output from "client" VisTrails<br />
*** Add client/server code to <tt>core</tt> that doesn't use Qt? Also need a run.py-line entrypoint that doesn't use gui (calls console mode like <tt>gui.application.VistrailsApplicationSingleton#noninteractiveMode()</tt>)?<br />
<br />
=== November 2, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* New version 2.2-2 of Debian package, still [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=796994 wrong copyright], and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=748692 broken bundle installation]<br />
** Finally replied, says copyright issue will be fixed soon, working on including VisTrails 2.2.4 for Debian Stretch<br />
* Working on Docker package, low hanging fruit because we have CLTools, and useful for some workflow (also step towards CWL support)<br />
<br />
'''Items to Discuss'''<br />
<br />
=== October 5, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
<br />
* working toward a 3.0 release; [https://github.com/VisTrails/VisTrails/projects/1 project board]<br />
** unique ids<br />
** bundles<br />
** GUI fixes<br />
** wrapping updates<br />
** python3<br />
** interpreter?<br />
<br />
* command-line input parameters to provenance<br />
** [DK] we don't create a persisted workflow from those input params, i think<br />
** [DK] these are already defined as aliases so we treat them differently form a normal input param<br />
** [DK] still have the potential problem of allowing a user to view the workflow as it was run with a given set of aliases<br />
** Do the vistrails variables values get stored in provenance? (what if values change?)<br />
<br />
=== September 14, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] [http://aosabook.org/en/vistrails.html VisTrails] in [http://aosabook.org/ The Architecture of Open Source Applications] Vol I (2012)<br />
** Did we know/contribute to this?<br />
*** [DK] Yes. We wrote the text for the VisTrails chapter<br />
** Drafted response<br />
*** [DK] Looks good, will forward.<br />
<br />
=== August 31, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Went over vgc.poly.edu/vistrails.org config:<br />
** Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)<br />
** Fixed usersguide scripts (won't disappear again)<br />
** Moved some websites over to HTTPS<br />
** [https://docs.google.com/document/d/18YZewjR2YY9cQ9pzdA-nU9jbu2L7jYcPlDi3iaPadwA/edit Websites]<br />
* [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs ([http://vistrails.readthedocs.io/ currently there]) and users' guide under the same RTD "project"<br />
* [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that<br />
<br />
'''Items to Discuss'''<br />
* [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/<br />
** Get rid of more services? (svn, trac, ...)<br />
* [RR] Notebook execution works. How do we support more languages (than Python)?<br />
** Need: detection, metadata reader, code to input values<br />
* [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?<br />
<br />
=== August 17, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Fixed [https://github.com/VisTrails/VisTrails/commit/2cd42d156243067a7663bb0bf0156e5d87188c2f parameter order upgrade bug] that breaks VTK examples.<br />
* [TE] Matplotlib and sklearn are exportable using module-to-script api<br />
* [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.<br />
<br />
'''Items to Discuss'''<br />
* Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!<br />
* Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)<br />
* [TE] Wrap-up notes<br />
<br />
=== August 10, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Python Wrapper and scripting<br />
** All changes merged into python-wrapper branch<br />
*** Tests pass<br />
*** Still need to update sub-branches<br />
** Minor fixes<br />
*** vtkPlot3DReader upgrade bug<br />
*** Added tuple exporting<br />
** Most of vtk examples can be exported [https://gist.github.com/rexissimus/5ac7374cde8525d84f9443f019d234e2 example]<br />
*** Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary<br />
**** Could be a bug in the old vistrail, but not sure<br />
* [DK] Subworkflows and uuids<br />
<br />
=== August 3, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Wrapper patches now stored as [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/vtk_patch.py python code]<br />
** Connection variables now lowercase<br />
** Can we use code as output variables?<br />
*** E.g. "vtk_dataset_mapper.GetOutput(0)"<br />
*** Could mutate output when output method is only supposed to be called once<br />
*** Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)<br />
<br />
=== July 27, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] DB bundle serialization implemented<br />
<br />
'''Items to Discuss'''<br />
* [TE] Finished using patch in script [https://github.com/VisTrails/VisTrails/tree/python-wrapper-script-patches python-wrapper-script-patches]<br />
** locale?<br />
** [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d new patching example] vs [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 old patching example]<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/parse.py#L135-L273 Example of new patches] <br />
** Legacy '.vtkInstance' is now added as a patch "self.vtkInstance=self" (see new patching example)<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/api.py New api for executing standalone modules]<br />
*** Accessing module was disabled for core/api, but a different api for this is probably better anyway.<br />
*** api.Package now gives [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d#file-new_patching-py-L26 directly callable module compute functions]<br />
*** Only works on modules not using interpreter functionality ('.interpreter', etc.)<br />
**** for zip file modules and others that need temp files and use the interpreter for this, need to modify the module, but could work around with python libraries for managing temp files<br />
** Fixed more renaming issues - [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/scripts.py#L63-L64 function argument names should not be replaced]<br />
** Still need to do something nicer for the patch formatting<br />
*** patch keys currently contain non-python-variable characters but this can be fixed<br />
*** Need to replace string template with python variables that is replacable using RedBaron<br />
**** Patches could then look like:<br />
<br />
def vtkInstance():<br />
self.vtkInstance = self<br />
<br />
=== July 20, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] New patch method for python-wrapper [https://gist.github.com/rexissimus/009b303e4cadcd4b8f2fec6cf848cb2b example]<br />
** A patch is a string template with self/input/output variables for patching a class method <br />
** Patches are stored in xml specification<br />
** Patches are assigned to methods in ClassSpec's<br />
** Used both for execution in vistrails and when exporting as script<br />
*** Executed with "exec" when run in vistrails<br />
*** Turns patches into natural calls in script<br />
** Wrapping VTK library no longer needed<br />
*** Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)<br />
** Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)<br />
** Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)<br />
** TODO's<br />
*** Add to export_as_script<br />
*** Patch Modules that represent Python functions as well<br />
<br />
* [vistrails-users] Read data from SQL example?<br />
<br />
=== July 13, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] bundles on sql-alchemy branch<br />
** Bundle, BundleMapping, and BundleSerializer<br />
** New BaseSerializers to deal with versions of directory, zip, db serializers<br />
** Can register the BundleSerializers with the base serializers<br />
** Translation between bundle versions<br />
** Bundle.add_object "just works" if an appropriate mapping was registered<br />
<br />
'''Items to Discuss'''<br />
* [TE] Simplified running module using API for scripting [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 example]<br />
** API Module can be executed using '''compute''' method<br />
*** Takes list of outputs to compute + inputs as keyword arguments<br />
* Status of general cell synchronization (vistrails-users)<br />
<br />
=== July 6, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] Working on bundles<br />
<br />
'''Items to Discuss'''<br />
* [TE] Using vistrails API in scripting [https://gist.github.com/rexissimus/034d9c4dedc3299a079e84981dedd8c7 example 1 (API with VTK)] [https://gist.github.com/rexissimus/a4a9ca6c08d4058c64967d2b35415fd5 example 2 (DownloadFile only)]<br />
** Call module in script using vistrails API<br />
** Can be used when no export method is defined.<br />
* [TE] [https://github.com/VisTrails/VisTrails/commit/c76c109349769fee1280f85c22678380a2f9cd39 Server news bug] Critical?<br />
**[RR] No, unreleased<br />
* [DK] New bundle organization<br />
** BundleMapping contains individual mappings from raw objects to BundleObj<br />
** Must be separate in order to have in-memory, separate from serializer settings<br />
<br />
=== June 29, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as python script<br />
** Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) [https://gist.github.com/rexissimus/16caf5200802e150296f62a0c7844c5a example]<br />
** Preludes (imports) still need improvements<br />
** Works with vtkInteractionHandler (clipping plane)<br />
** Resulting clipping plane is incorrect? (supposed to hide one side of the plane)<br />
** vtkRendererOutput Uses VTKRenderWindow as output<br />
*** Should support different output modes?<br />
*** Does not work with multiple outputs (Next cell is displayed when current is closed)<br />
*** Uses wrapped vtkInstance (as vvtk)<br />
**** Create new patch system with code that can be used both in vistrails and in a python script?<br />
<br />
=== June 22, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fix for upgraded analogies (PR [https://github.com/VisTrails/VisTrails/pull/1177 #1177])<br />
** Uses module location to find module remap (The best we can do right now?)<br />
** Also fixes remapped port names for connections to existing modules <br />
** May still fail on complex upgrades like adding a function on an upgraded module that has a different port name<br />
<br />
* [TE] Working on exporting wrapped libraries (VTK)<br />
<br />
=== June 15, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Added depth logic to python script exporter ([https://github.com/VisTrails/VisTrails/commit/50f220e4f6abe5a7bb8689f73f456f43df2dbc12 full commit])<br />
** Fully implements connection merging and list looping<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Making analogy work on upgraded workflows ([https://github.com/VisTrails/VisTrails/issues/1175 #1175])<br />
** Analogy only works We need to use the non-upgraded action chain<br />
** But pipelines need to be valid (matcher uses module portspecs)<br />
** Tried to get upgrade remap from upgrade action, but this seems impossible<br />
** Use module location instead?<br />
*** Location should be identical for one-to-one module upgrades<br />
<br />
=== June 1, 2016 ===<br />
<br />
'''Updates'''<br />
* VisTrails 2.2.4 announced (finally)<br />
** 5 survey replies<br />
** 47 downloads last week<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as script (python-wrapper)<br />
** to_python_script<br />
*** needs the pipeline (or at least the connections)<br />
**** Should not generate code for any possible VTK method (exporter detects which ports is used in the source)<br />
**** VTK needs to know connection order<br />
**** Adding functions directly to the script results in nicer code<br />
** Add support for multiple input connections<br />
** Add support for list looping<br />
<br />
=== May 25, 2016 ===<br />
<br />
'''Updates'''<br />
* Mailing-list fixed<br />
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])<br />
<br />
'''Items to Discuss'''<br />
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)<br />
* [TE] Union ports<br />
** Show union ports in pipeline (DEMO)<br />
<br />
=== May 18, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] mailing list still down - Julio working on it<br />
* [TE] Improved union port<br />
** New schema 1.0.5 adds portitem.union string attribute<br />
** Works with pipeline view, parameter exploration view, and mashup view<br />
** Should work well for matplotlib<br />
** Use with VTK ports?<br />
*** It uses [item_1,item_2] style ports usually with different tuple length.<br />
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]<br />
<br />
=== May 11, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?<br />
*** Shows single port for all ports with same sort_key<br />
*** Select specific type when adding/showing<br />
*** Only one type should be visible on the module at any time<br />
*** Making tuple parameters union types is probably too complicated<br />
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly) <br />
<br />
* [TE] Feature suggestion: Let connections connect to functions/parameters<br />
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters<br />
** Use separate views for ports and functions (like before)<br />
*** Allow rearranging of functions<br />
** Pros:<br />
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.<br />
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.<br />
**** Makes List module simpler<br />
*** Connect to parameter in tuple<br />
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds. <br />
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.<br />
** Cons:<br />
*** Interface more complicated and crowded<br />
*** More difficult for users to grasp?<br />
*** No immediate use case<br />
*** Makes no union type for tuple parameters a bigger problem.<br />
<br />
* [TE] Publish user survey?<br />
** No response from vistrails release yet<br />
** Not announced on vistrails-users yet.<br />
<br />
=== May 4, 2016 ===<br />
'''Updates'''<br />
* VisTrails 2.2.4 released<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
<br />
=== April 27, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Wrapping Bokeh<br />
** New method type "operation"<br />
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output<br />
**** Glyph sets function name and arguments as output<br />
***** Not serializable, but should always be computed together with the subject of the operation.<br />
**** Figure applies Glyph operation on itself<br />
*** Useful for matplotlib?<br />
**** matplotlib currently find actors by searching upstream?<br />
**** But this is more difficult since matplotlib is not function-based.<br />
** New "union" port type<br />
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value<br />
**** Uses only first value in tuple<br />
**** Bokeh have many properties that support (list, string, Float)<br />
*** Does not work with depth=1 types (e.g. Float of depth 1)<br />
**** Turn Depth 1 types with known type into lists?<br />
<br />
* [TE] New bugfix release<br />
** Finalize survey<br />
<br />
=== April 20, 2016 ===<br />
'''Updates'''<br />
* Merged upgrade fixes<br />
** Now has infinite loop test<br />
** loop test now correctly loops 5 times<br />
*** remap_module used wrong (current) package version when creating ModuleDescriptor<br />
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])<br />
<br />
'''Items to Discuss'''<br />
* New bugfix release<br />
** Merge usagestats<br />
** Need to finalize survey<br />
<br />
=== April 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?<br />
<br />
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])<br />
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]<br />
** Cherry-pick test to preserve source?<br />
** Need maxPipelineFixAttempts option?<br />
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.<br />
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?<br />
*** Would need to be moved into handle_invalid_pipeline<br />
<br />
=== April 6, 2016 ===<br />
'''Updates'''<br />
* [TE] Query View improvements<br />
** Done except for displaying "Version Info" and #1164<br />
* [RR] Working on finishing new interpreter (finally!)<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Try latest upgrade first, then previous.<br />
** If all fail, select latest upgrade?<br />
<br />
=== March 30, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Query View improvements<br />
** Unified do_version_switch and get_upgrades as validate_version()<br />
** Problems with query view using its own controller<br />
*** Fixed module info not being displayed for query results<br />
*** Still no version info for query results<br />
*** Still an issue with version query view not being updated<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== Older meetings ==<br />
* [[Development/2015]]<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=13329Development2017-02-24T18:45:21Z<p>Remi: /* February 24, 2017 */</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2017 ==<br />
=== February 24, 2017 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
=== January 25, 2017 ===<br />
<br />
'''Updates'''<br />
<br />
<br />
'''Items to Discuss'''<br />
<br />
* Scaling VisTrails<br />
* sql-alchemy, uuid, bundle branches/merges<br />
<br />
=== January 11, 2017 ===<br />
<br />
'''Updates'''<br />
* [RR] New VisTrails version in Debian [https://packages.qa.debian.org/v/vistrails.html]<br />
<br />
'''Items to Discuss'''<br />
* [RR] Collaboration with school of medicine to port HiC-bench to VisTrails<br />
** Have homemade workflow system based on R scripts and symlinks<br />
*** Each steps has multiple sets of parameters<br />
*** Submits PBS jobs for each step with each set of parameters, using PBS job metadata to wait on previous step<br />
*** Gets all results in directory structure <code>step3-parameters/step2-parameters/step1-parameters/files</code><br />
** VisTrails improves provenance capture and makes editing pipelines easier<br />
*** Need better support for aggregation (groupby operation might be good enough; pandas?)<br />
<br />
== 2016 ==<br />
=== December 14, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Single-instance code: over-engineered, but will work<br />
** (Have to avoid race conditions on startup, and OSes offer very different locking primitives)<br />
<br />
'''Items to Discuss'''<br />
* Email from Matt Dirks: combobox should work, ask for more code?<br />
* Email from marin.nl:<br />
** looks like they would benefit from new interpreter<br />
** how to store their consolidated provenance when they exchange .vt files around?<br />
<br />
=== November 30, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Working on single instance code for MARIN ([https://github.com/VisTrails/VisTrails/pull/1205 #1205])<br />
** Remote shutdown ready ([https://github.com/VisTrails/VisTrails/pull/1204 #1204])<br />
** Still improving it<br />
** Want to return something more structured to the client (success/fail code, stdout, stderr)<br />
** Should be usable without Qt<br />
* [DK] Shutting down VisTrails has issues: cleans up interpreter (to remove files) but also resets persistent pipeline for no reason<br />
* [DK] Working on batch execution using console_mode<br />
* [DK] Settings aliased parameters when using core.api?<br />
<br />
=== November 16, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [RR] Issue [https://github.com/VisTrails/VisTrails/issues/1200 #1200] from @samtux: need to set PyQt to API version 2 before ever importing it; using require_pyqt_api2() does that (also can install bundle)<br />
<br />
'''Items to Discuss'''<br />
<br />
* Email from Pieter: executing pipelines with parameters from command-line using single-instance<br />
** Seems to work, main instance gets full command-line and alters configuration temporarily<br />
** Not sure what his error is (VisTrails returns little information in a lot of cases); though execution error should return something more meaningful<br />
** Actions:<br />
*** Can add switch to kill main instance<br />
*** Can improve single-instance code to return more meaningful errors + machine-readable output from "client" VisTrails<br />
*** Add client/server code to <tt>core</tt> that doesn't use Qt? Also need a run.py-line entrypoint that doesn't use gui (calls console mode like <tt>gui.application.VistrailsApplicationSingleton#noninteractiveMode()</tt>)?<br />
<br />
=== November 2, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* New version 2.2-2 of Debian package, still [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=796994 wrong copyright], and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=748692 broken bundle installation]<br />
** Finally replied, says copyright issue will be fixed soon, working on including VisTrails 2.2.4 for Debian Stretch<br />
* Working on Docker package, low hanging fruit because we have CLTools, and useful for some workflow (also step towards CWL support)<br />
<br />
'''Items to Discuss'''<br />
<br />
=== October 5, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
<br />
* working toward a 3.0 release; [https://github.com/VisTrails/VisTrails/projects/1 project board]<br />
** unique ids<br />
** bundles<br />
** GUI fixes<br />
** wrapping updates<br />
** python3<br />
** interpreter?<br />
<br />
* command-line input parameters to provenance<br />
** [DK] we don't create a persisted workflow from those input params, i think<br />
** [DK] these are already defined as aliases so we treat them differently form a normal input param<br />
** [DK] still have the potential problem of allowing a user to view the workflow as it was run with a given set of aliases<br />
** Do the vistrails variables values get stored in provenance? (what if values change?)<br />
<br />
=== September 14, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] [http://aosabook.org/en/vistrails.html VisTrails] in [http://aosabook.org/ The Architecture of Open Source Applications] Vol I (2012)<br />
** Did we know/contribute to this?<br />
*** [DK] Yes. We wrote the text for the VisTrails chapter<br />
** Drafted response<br />
*** [DK] Looks good, will forward.<br />
<br />
=== August 31, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Went over vgc.poly.edu/vistrails.org config:<br />
** Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)<br />
** Fixed usersguide scripts (won't disappear again)<br />
** Moved some websites over to HTTPS<br />
** [https://docs.google.com/document/d/18YZewjR2YY9cQ9pzdA-nU9jbu2L7jYcPlDi3iaPadwA/edit Websites]<br />
* [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs ([http://vistrails.readthedocs.io/ currently there]) and users' guide under the same RTD "project"<br />
* [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that<br />
<br />
'''Items to Discuss'''<br />
* [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/<br />
** Get rid of more services? (svn, trac, ...)<br />
* [RR] Notebook execution works. How do we support more languages (than Python)?<br />
** Need: detection, metadata reader, code to input values<br />
* [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?<br />
<br />
=== August 17, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Fixed [https://github.com/VisTrails/VisTrails/commit/2cd42d156243067a7663bb0bf0156e5d87188c2f parameter order upgrade bug] that breaks VTK examples.<br />
* [TE] Matplotlib and sklearn are exportable using module-to-script api<br />
* [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.<br />
<br />
'''Items to Discuss'''<br />
* Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!<br />
* Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)<br />
* [TE] Wrap-up notes<br />
<br />
=== August 10, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Python Wrapper and scripting<br />
** All changes merged into python-wrapper branch<br />
*** Tests pass<br />
*** Still need to update sub-branches<br />
** Minor fixes<br />
*** vtkPlot3DReader upgrade bug<br />
*** Added tuple exporting<br />
** Most of vtk examples can be exported [https://gist.github.com/rexissimus/5ac7374cde8525d84f9443f019d234e2 example]<br />
*** Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary<br />
**** Could be a bug in the old vistrail, but not sure<br />
* [DK] Subworkflows and uuids<br />
<br />
=== August 3, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Wrapper patches now stored as [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/vtk_patch.py python code]<br />
** Connection variables now lowercase<br />
** Can we use code as output variables?<br />
*** E.g. "vtk_dataset_mapper.GetOutput(0)"<br />
*** Could mutate output when output method is only supposed to be called once<br />
*** Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)<br />
<br />
=== July 27, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] DB bundle serialization implemented<br />
<br />
'''Items to Discuss'''<br />
* [TE] Finished using patch in script [https://github.com/VisTrails/VisTrails/tree/python-wrapper-script-patches python-wrapper-script-patches]<br />
** locale?<br />
** [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d new patching example] vs [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 old patching example]<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/parse.py#L135-L273 Example of new patches] <br />
** Legacy '.vtkInstance' is now added as a patch "self.vtkInstance=self" (see new patching example)<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/api.py New api for executing standalone modules]<br />
*** Accessing module was disabled for core/api, but a different api for this is probably better anyway.<br />
*** api.Package now gives [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d#file-new_patching-py-L26 directly callable module compute functions]<br />
*** Only works on modules not using interpreter functionality ('.interpreter', etc.)<br />
**** for zip file modules and others that need temp files and use the interpreter for this, need to modify the module, but could work around with python libraries for managing temp files<br />
** Fixed more renaming issues - [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/scripts.py#L63-L64 function argument names should not be replaced]<br />
** Still need to do something nicer for the patch formatting<br />
*** patch keys currently contain non-python-variable characters but this can be fixed<br />
*** Need to replace string template with python variables that is replacable using RedBaron<br />
**** Patches could then look like:<br />
<br />
def vtkInstance():<br />
self.vtkInstance = self<br />
<br />
=== July 20, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] New patch method for python-wrapper [https://gist.github.com/rexissimus/009b303e4cadcd4b8f2fec6cf848cb2b example]<br />
** A patch is a string template with self/input/output variables for patching a class method <br />
** Patches are stored in xml specification<br />
** Patches are assigned to methods in ClassSpec's<br />
** Used both for execution in vistrails and when exporting as script<br />
*** Executed with "exec" when run in vistrails<br />
*** Turns patches into natural calls in script<br />
** Wrapping VTK library no longer needed<br />
*** Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)<br />
** Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)<br />
** Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)<br />
** TODO's<br />
*** Add to export_as_script<br />
*** Patch Modules that represent Python functions as well<br />
<br />
* [vistrails-users] Read data from SQL example?<br />
<br />
=== July 13, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] bundles on sql-alchemy branch<br />
** Bundle, BundleMapping, and BundleSerializer<br />
** New BaseSerializers to deal with versions of directory, zip, db serializers<br />
** Can register the BundleSerializers with the base serializers<br />
** Translation between bundle versions<br />
** Bundle.add_object "just works" if an appropriate mapping was registered<br />
<br />
'''Items to Discuss'''<br />
* [TE] Simplified running module using API for scripting [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 example]<br />
** API Module can be executed using '''compute''' method<br />
*** Takes list of outputs to compute + inputs as keyword arguments<br />
* Status of general cell synchronization (vistrails-users)<br />
<br />
=== July 6, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] Working on bundles<br />
<br />
'''Items to Discuss'''<br />
* [TE] Using vistrails API in scripting [https://gist.github.com/rexissimus/034d9c4dedc3299a079e84981dedd8c7 example 1 (API with VTK)] [https://gist.github.com/rexissimus/a4a9ca6c08d4058c64967d2b35415fd5 example 2 (DownloadFile only)]<br />
** Call module in script using vistrails API<br />
** Can be used when no export method is defined.<br />
* [TE] [https://github.com/VisTrails/VisTrails/commit/c76c109349769fee1280f85c22678380a2f9cd39 Server news bug] Critical?<br />
**[RR] No, unreleased<br />
* [DK] New bundle organization<br />
** BundleMapping contains individual mappings from raw objects to BundleObj<br />
** Must be separate in order to have in-memory, separate from serializer settings<br />
<br />
=== June 29, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as python script<br />
** Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) [https://gist.github.com/rexissimus/16caf5200802e150296f62a0c7844c5a example]<br />
** Preludes (imports) still need improvements<br />
** Works with vtkInteractionHandler (clipping plane)<br />
** Resulting clipping plane is incorrect? (supposed to hide one side of the plane)<br />
** vtkRendererOutput Uses VTKRenderWindow as output<br />
*** Should support different output modes?<br />
*** Does not work with multiple outputs (Next cell is displayed when current is closed)<br />
*** Uses wrapped vtkInstance (as vvtk)<br />
**** Create new patch system with code that can be used both in vistrails and in a python script?<br />
<br />
=== June 22, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fix for upgraded analogies (PR [https://github.com/VisTrails/VisTrails/pull/1177 #1177])<br />
** Uses module location to find module remap (The best we can do right now?)<br />
** Also fixes remapped port names for connections to existing modules <br />
** May still fail on complex upgrades like adding a function on an upgraded module that has a different port name<br />
<br />
* [TE] Working on exporting wrapped libraries (VTK)<br />
<br />
=== June 15, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Added depth logic to python script exporter ([https://github.com/VisTrails/VisTrails/commit/50f220e4f6abe5a7bb8689f73f456f43df2dbc12 full commit])<br />
** Fully implements connection merging and list looping<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Making analogy work on upgraded workflows ([https://github.com/VisTrails/VisTrails/issues/1175 #1175])<br />
** Analogy only works We need to use the non-upgraded action chain<br />
** But pipelines need to be valid (matcher uses module portspecs)<br />
** Tried to get upgrade remap from upgrade action, but this seems impossible<br />
** Use module location instead?<br />
*** Location should be identical for one-to-one module upgrades<br />
<br />
=== June 1, 2016 ===<br />
<br />
'''Updates'''<br />
* VisTrails 2.2.4 announced (finally)<br />
** 5 survey replies<br />
** 47 downloads last week<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as script (python-wrapper)<br />
** to_python_script<br />
*** needs the pipeline (or at least the connections)<br />
**** Should not generate code for any possible VTK method (exporter detects which ports is used in the source)<br />
**** VTK needs to know connection order<br />
**** Adding functions directly to the script results in nicer code<br />
** Add support for multiple input connections<br />
** Add support for list looping<br />
<br />
=== May 25, 2016 ===<br />
<br />
'''Updates'''<br />
* Mailing-list fixed<br />
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])<br />
<br />
'''Items to Discuss'''<br />
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)<br />
* [TE] Union ports<br />
** Show union ports in pipeline (DEMO)<br />
<br />
=== May 18, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] mailing list still down - Julio working on it<br />
* [TE] Improved union port<br />
** New schema 1.0.5 adds portitem.union string attribute<br />
** Works with pipeline view, parameter exploration view, and mashup view<br />
** Should work well for matplotlib<br />
** Use with VTK ports?<br />
*** It uses [item_1,item_2] style ports usually with different tuple length.<br />
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]<br />
<br />
=== May 11, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?<br />
*** Shows single port for all ports with same sort_key<br />
*** Select specific type when adding/showing<br />
*** Only one type should be visible on the module at any time<br />
*** Making tuple parameters union types is probably too complicated<br />
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly) <br />
<br />
* [TE] Feature suggestion: Let connections connect to functions/parameters<br />
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters<br />
** Use separate views for ports and functions (like before)<br />
*** Allow rearranging of functions<br />
** Pros:<br />
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.<br />
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.<br />
**** Makes List module simpler<br />
*** Connect to parameter in tuple<br />
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds. <br />
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.<br />
** Cons:<br />
*** Interface more complicated and crowded<br />
*** More difficult for users to grasp?<br />
*** No immediate use case<br />
*** Makes no union type for tuple parameters a bigger problem.<br />
<br />
* [TE] Publish user survey?<br />
** No response from vistrails release yet<br />
** Not announced on vistrails-users yet.<br />
<br />
=== May 4, 2016 ===<br />
'''Updates'''<br />
* VisTrails 2.2.4 released<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
<br />
=== April 27, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Wrapping Bokeh<br />
** New method type "operation"<br />
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output<br />
**** Glyph sets function name and arguments as output<br />
***** Not serializable, but should always be computed together with the subject of the operation.<br />
**** Figure applies Glyph operation on itself<br />
*** Useful for matplotlib?<br />
**** matplotlib currently find actors by searching upstream?<br />
**** But this is more difficult since matplotlib is not function-based.<br />
** New "union" port type<br />
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value<br />
**** Uses only first value in tuple<br />
**** Bokeh have many properties that support (list, string, Float)<br />
*** Does not work with depth=1 types (e.g. Float of depth 1)<br />
**** Turn Depth 1 types with known type into lists?<br />
<br />
* [TE] New bugfix release<br />
** Finalize survey<br />
<br />
=== April 20, 2016 ===<br />
'''Updates'''<br />
* Merged upgrade fixes<br />
** Now has infinite loop test<br />
** loop test now correctly loops 5 times<br />
*** remap_module used wrong (current) package version when creating ModuleDescriptor<br />
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])<br />
<br />
'''Items to Discuss'''<br />
* New bugfix release<br />
** Merge usagestats<br />
** Need to finalize survey<br />
<br />
=== April 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?<br />
<br />
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])<br />
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]<br />
** Cherry-pick test to preserve source?<br />
** Need maxPipelineFixAttempts option?<br />
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.<br />
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?<br />
*** Would need to be moved into handle_invalid_pipeline<br />
<br />
=== April 6, 2016 ===<br />
'''Updates'''<br />
* [TE] Query View improvements<br />
** Done except for displaying "Version Info" and #1164<br />
* [RR] Working on finishing new interpreter (finally!)<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Try latest upgrade first, then previous.<br />
** If all fail, select latest upgrade?<br />
<br />
=== March 30, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Query View improvements<br />
** Unified do_version_switch and get_upgrades as validate_version()<br />
** Problems with query view using its own controller<br />
*** Fixed module info not being displayed for query results<br />
*** Still no version info for query results<br />
*** Still an issue with version query view not being updated<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== Older meetings ==<br />
* [[Development/2015]]<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=13168Development2017-01-25T15:47:57Z<p>Remi: /* January 25, 2017 */</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2017 ==<br />
=== January 25, 2017 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
=== January 11, 2017 ===<br />
<br />
'''Updates'''<br />
* [RR] New VisTrails version in Debian [https://packages.qa.debian.org/v/vistrails.html]<br />
<br />
'''Items to Discuss'''<br />
* [RR] Collaboration with school of medicine to port HiC-bench to VisTrails<br />
** Have homemade workflow system based on R scripts and symlinks<br />
*** Each steps has multiple sets of parameters<br />
*** Submits PBS jobs for each step with each set of parameters, using PBS job metadata to wait on previous step<br />
*** Gets all results in directory structure <code>step3-parameters/step2-parameters/step1-parameters/files</code><br />
** VisTrails improves provenance capture and makes editing pipelines easier<br />
*** Need better support for aggregation (groupby operation might be good enough; pandas?)<br />
<br />
== 2016 ==<br />
=== December 14, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Single-instance code: over-engineered, but will work<br />
** (Have to avoid race conditions on startup, and OSes offer very different locking primitives)<br />
<br />
'''Items to Discuss'''<br />
* Email from Matt Dirks: combobox should work, ask for more code?<br />
* Email from marin.nl:<br />
** looks like they would benefit from new interpreter<br />
** how to store their consolidated provenance when they exchange .vt files around?<br />
<br />
=== November 30, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Working on single instance code for MARIN ([https://github.com/VisTrails/VisTrails/pull/1205 #1205])<br />
** Remote shutdown ready ([https://github.com/VisTrails/VisTrails/pull/1204 #1204])<br />
** Still improving it<br />
** Want to return something more structured to the client (success/fail code, stdout, stderr)<br />
** Should be usable without Qt<br />
* [DK] Shutting down VisTrails has issues: cleans up interpreter (to remove files) but also resets persistent pipeline for no reason<br />
* [DK] Working on batch execution using console_mode<br />
* [DK] Settings aliased parameters when using core.api?<br />
<br />
=== November 16, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [RR] Issue [https://github.com/VisTrails/VisTrails/issues/1200 #1200] from @samtux: need to set PyQt to API version 2 before ever importing it; using require_pyqt_api2() does that (also can install bundle)<br />
<br />
'''Items to Discuss'''<br />
<br />
* Email from Pieter: executing pipelines with parameters from command-line using single-instance<br />
** Seems to work, main instance gets full command-line and alters configuration temporarily<br />
** Not sure what his error is (VisTrails returns little information in a lot of cases); though execution error should return something more meaningful<br />
** Actions:<br />
*** Can add switch to kill main instance<br />
*** Can improve single-instance code to return more meaningful errors + machine-readable output from "client" VisTrails<br />
*** Add client/server code to <tt>core</tt> that doesn't use Qt? Also need a run.py-line entrypoint that doesn't use gui (calls console mode like <tt>gui.application.VistrailsApplicationSingleton#noninteractiveMode()</tt>)?<br />
<br />
=== November 2, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* New version 2.2-2 of Debian package, still [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=796994 wrong copyright], and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=748692 broken bundle installation]<br />
** Finally replied, says copyright issue will be fixed soon, working on including VisTrails 2.2.4 for Debian Stretch<br />
* Working on Docker package, low hanging fruit because we have CLTools, and useful for some workflow (also step towards CWL support)<br />
<br />
'''Items to Discuss'''<br />
<br />
=== October 5, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
<br />
* working toward a 3.0 release; [https://github.com/VisTrails/VisTrails/projects/1 project board]<br />
** unique ids<br />
** bundles<br />
** GUI fixes<br />
** wrapping updates<br />
** python3<br />
** interpreter?<br />
<br />
* command-line input parameters to provenance<br />
** [DK] we don't create a persisted workflow from those input params, i think<br />
** [DK] these are already defined as aliases so we treat them differently form a normal input param<br />
** [DK] still have the potential problem of allowing a user to view the workflow as it was run with a given set of aliases<br />
** Do the vistrails variables values get stored in provenance? (what if values change?)<br />
<br />
=== September 14, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] [http://aosabook.org/en/vistrails.html VisTrails] in [http://aosabook.org/ The Architecture of Open Source Applications] Vol I (2012)<br />
** Did we know/contribute to this?<br />
*** [DK] Yes. We wrote the text for the VisTrails chapter<br />
** Drafted response<br />
*** [DK] Looks good, will forward.<br />
<br />
=== August 31, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Went over vgc.poly.edu/vistrails.org config:<br />
** Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)<br />
** Fixed usersguide scripts (won't disappear again)<br />
** Moved some websites over to HTTPS<br />
** [https://docs.google.com/document/d/18YZewjR2YY9cQ9pzdA-nU9jbu2L7jYcPlDi3iaPadwA/edit Websites]<br />
* [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs ([http://vistrails.readthedocs.io/ currently there]) and users' guide under the same RTD "project"<br />
* [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that<br />
<br />
'''Items to Discuss'''<br />
* [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/<br />
** Get rid of more services? (svn, trac, ...)<br />
* [RR] Notebook execution works. How do we support more languages (than Python)?<br />
** Need: detection, metadata reader, code to input values<br />
* [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?<br />
<br />
=== August 17, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Fixed [https://github.com/VisTrails/VisTrails/commit/2cd42d156243067a7663bb0bf0156e5d87188c2f parameter order upgrade bug] that breaks VTK examples.<br />
* [TE] Matplotlib and sklearn are exportable using module-to-script api<br />
* [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.<br />
<br />
'''Items to Discuss'''<br />
* Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!<br />
* Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)<br />
* [TE] Wrap-up notes<br />
<br />
=== August 10, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Python Wrapper and scripting<br />
** All changes merged into python-wrapper branch<br />
*** Tests pass<br />
*** Still need to update sub-branches<br />
** Minor fixes<br />
*** vtkPlot3DReader upgrade bug<br />
*** Added tuple exporting<br />
** Most of vtk examples can be exported [https://gist.github.com/rexissimus/5ac7374cde8525d84f9443f019d234e2 example]<br />
*** Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary<br />
**** Could be a bug in the old vistrail, but not sure<br />
* [DK] Subworkflows and uuids<br />
<br />
=== August 3, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Wrapper patches now stored as [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/vtk_patch.py python code]<br />
** Connection variables now lowercase<br />
** Can we use code as output variables?<br />
*** E.g. "vtk_dataset_mapper.GetOutput(0)"<br />
*** Could mutate output when output method is only supposed to be called once<br />
*** Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)<br />
<br />
=== July 27, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] DB bundle serialization implemented<br />
<br />
'''Items to Discuss'''<br />
* [TE] Finished using patch in script [https://github.com/VisTrails/VisTrails/tree/python-wrapper-script-patches python-wrapper-script-patches]<br />
** locale?<br />
** [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d new patching example] vs [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 old patching example]<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/parse.py#L135-L273 Example of new patches] <br />
** Legacy '.vtkInstance' is now added as a patch "self.vtkInstance=self" (see new patching example)<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/api.py New api for executing standalone modules]<br />
*** Accessing module was disabled for core/api, but a different api for this is probably better anyway.<br />
*** api.Package now gives [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d#file-new_patching-py-L26 directly callable module compute functions]<br />
*** Only works on modules not using interpreter functionality ('.interpreter', etc.)<br />
**** for zip file modules and others that need temp files and use the interpreter for this, need to modify the module, but could work around with python libraries for managing temp files<br />
** Fixed more renaming issues - [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/scripts.py#L63-L64 function argument names should not be replaced]<br />
** Still need to do something nicer for the patch formatting<br />
*** patch keys currently contain non-python-variable characters but this can be fixed<br />
*** Need to replace string template with python variables that is replacable using RedBaron<br />
**** Patches could then look like:<br />
<br />
def vtkInstance():<br />
self.vtkInstance = self<br />
<br />
=== July 20, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] New patch method for python-wrapper [https://gist.github.com/rexissimus/009b303e4cadcd4b8f2fec6cf848cb2b example]<br />
** A patch is a string template with self/input/output variables for patching a class method <br />
** Patches are stored in xml specification<br />
** Patches are assigned to methods in ClassSpec's<br />
** Used both for execution in vistrails and when exporting as script<br />
*** Executed with "exec" when run in vistrails<br />
*** Turns patches into natural calls in script<br />
** Wrapping VTK library no longer needed<br />
*** Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)<br />
** Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)<br />
** Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)<br />
** TODO's<br />
*** Add to export_as_script<br />
*** Patch Modules that represent Python functions as well<br />
<br />
* [vistrails-users] Read data from SQL example?<br />
<br />
=== July 13, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] bundles on sql-alchemy branch<br />
** Bundle, BundleMapping, and BundleSerializer<br />
** New BaseSerializers to deal with versions of directory, zip, db serializers<br />
** Can register the BundleSerializers with the base serializers<br />
** Translation between bundle versions<br />
** Bundle.add_object "just works" if an appropriate mapping was registered<br />
<br />
'''Items to Discuss'''<br />
* [TE] Simplified running module using API for scripting [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 example]<br />
** API Module can be executed using '''compute''' method<br />
*** Takes list of outputs to compute + inputs as keyword arguments<br />
* Status of general cell synchronization (vistrails-users)<br />
<br />
=== July 6, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] Working on bundles<br />
<br />
'''Items to Discuss'''<br />
* [TE] Using vistrails API in scripting [https://gist.github.com/rexissimus/034d9c4dedc3299a079e84981dedd8c7 example 1 (API with VTK)] [https://gist.github.com/rexissimus/a4a9ca6c08d4058c64967d2b35415fd5 example 2 (DownloadFile only)]<br />
** Call module in script using vistrails API<br />
** Can be used when no export method is defined.<br />
* [TE] [https://github.com/VisTrails/VisTrails/commit/c76c109349769fee1280f85c22678380a2f9cd39 Server news bug] Critical?<br />
**[RR] No, unreleased<br />
* [DK] New bundle organization<br />
** BundleMapping contains individual mappings from raw objects to BundleObj<br />
** Must be separate in order to have in-memory, separate from serializer settings<br />
<br />
=== June 29, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as python script<br />
** Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) [https://gist.github.com/rexissimus/16caf5200802e150296f62a0c7844c5a example]<br />
** Preludes (imports) still need improvements<br />
** Works with vtkInteractionHandler (clipping plane)<br />
** Resulting clipping plane is incorrect? (supposed to hide one side of the plane)<br />
** vtkRendererOutput Uses VTKRenderWindow as output<br />
*** Should support different output modes?<br />
*** Does not work with multiple outputs (Next cell is displayed when current is closed)<br />
*** Uses wrapped vtkInstance (as vvtk)<br />
**** Create new patch system with code that can be used both in vistrails and in a python script?<br />
<br />
=== June 22, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fix for upgraded analogies (PR [https://github.com/VisTrails/VisTrails/pull/1177 #1177])<br />
** Uses module location to find module remap (The best we can do right now?)<br />
** Also fixes remapped port names for connections to existing modules <br />
** May still fail on complex upgrades like adding a function on an upgraded module that has a different port name<br />
<br />
* [TE] Working on exporting wrapped libraries (VTK)<br />
<br />
=== June 15, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Added depth logic to python script exporter ([https://github.com/VisTrails/VisTrails/commit/50f220e4f6abe5a7bb8689f73f456f43df2dbc12 full commit])<br />
** Fully implements connection merging and list looping<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Making analogy work on upgraded workflows ([https://github.com/VisTrails/VisTrails/issues/1175 #1175])<br />
** Analogy only works We need to use the non-upgraded action chain<br />
** But pipelines need to be valid (matcher uses module portspecs)<br />
** Tried to get upgrade remap from upgrade action, but this seems impossible<br />
** Use module location instead?<br />
*** Location should be identical for one-to-one module upgrades<br />
<br />
=== June 1, 2016 ===<br />
<br />
'''Updates'''<br />
* VisTrails 2.2.4 announced (finally)<br />
** 5 survey replies<br />
** 47 downloads last week<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as script (python-wrapper)<br />
** to_python_script<br />
*** needs the pipeline (or at least the connections)<br />
**** Should not generate code for any possible VTK method (exporter detects which ports is used in the source)<br />
**** VTK needs to know connection order<br />
**** Adding functions directly to the script results in nicer code<br />
** Add support for multiple input connections<br />
** Add support for list looping<br />
<br />
=== May 25, 2016 ===<br />
<br />
'''Updates'''<br />
* Mailing-list fixed<br />
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])<br />
<br />
'''Items to Discuss'''<br />
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)<br />
* [TE] Union ports<br />
** Show union ports in pipeline (DEMO)<br />
<br />
=== May 18, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] mailing list still down - Julio working on it<br />
* [TE] Improved union port<br />
** New schema 1.0.5 adds portitem.union string attribute<br />
** Works with pipeline view, parameter exploration view, and mashup view<br />
** Should work well for matplotlib<br />
** Use with VTK ports?<br />
*** It uses [item_1,item_2] style ports usually with different tuple length.<br />
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]<br />
<br />
=== May 11, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?<br />
*** Shows single port for all ports with same sort_key<br />
*** Select specific type when adding/showing<br />
*** Only one type should be visible on the module at any time<br />
*** Making tuple parameters union types is probably too complicated<br />
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly) <br />
<br />
* [TE] Feature suggestion: Let connections connect to functions/parameters<br />
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters<br />
** Use separate views for ports and functions (like before)<br />
*** Allow rearranging of functions<br />
** Pros:<br />
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.<br />
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.<br />
**** Makes List module simpler<br />
*** Connect to parameter in tuple<br />
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds. <br />
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.<br />
** Cons:<br />
*** Interface more complicated and crowded<br />
*** More difficult for users to grasp?<br />
*** No immediate use case<br />
*** Makes no union type for tuple parameters a bigger problem.<br />
<br />
* [TE] Publish user survey?<br />
** No response from vistrails release yet<br />
** Not announced on vistrails-users yet.<br />
<br />
=== May 4, 2016 ===<br />
'''Updates'''<br />
* VisTrails 2.2.4 released<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
<br />
=== April 27, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Wrapping Bokeh<br />
** New method type "operation"<br />
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output<br />
**** Glyph sets function name and arguments as output<br />
***** Not serializable, but should always be computed together with the subject of the operation.<br />
**** Figure applies Glyph operation on itself<br />
*** Useful for matplotlib?<br />
**** matplotlib currently find actors by searching upstream?<br />
**** But this is more difficult since matplotlib is not function-based.<br />
** New "union" port type<br />
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value<br />
**** Uses only first value in tuple<br />
**** Bokeh have many properties that support (list, string, Float)<br />
*** Does not work with depth=1 types (e.g. Float of depth 1)<br />
**** Turn Depth 1 types with known type into lists?<br />
<br />
* [TE] New bugfix release<br />
** Finalize survey<br />
<br />
=== April 20, 2016 ===<br />
'''Updates'''<br />
* Merged upgrade fixes<br />
** Now has infinite loop test<br />
** loop test now correctly loops 5 times<br />
*** remap_module used wrong (current) package version when creating ModuleDescriptor<br />
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])<br />
<br />
'''Items to Discuss'''<br />
* New bugfix release<br />
** Merge usagestats<br />
** Need to finalize survey<br />
<br />
=== April 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?<br />
<br />
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])<br />
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]<br />
** Cherry-pick test to preserve source?<br />
** Need maxPipelineFixAttempts option?<br />
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.<br />
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?<br />
*** Would need to be moved into handle_invalid_pipeline<br />
<br />
=== April 6, 2016 ===<br />
'''Updates'''<br />
* [TE] Query View improvements<br />
** Done except for displaying "Version Info" and #1164<br />
* [RR] Working on finishing new interpreter (finally!)<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Try latest upgrade first, then previous.<br />
** If all fail, select latest upgrade?<br />
<br />
=== March 30, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Query View improvements<br />
** Unified do_version_switch and get_upgrades as validate_version()<br />
** Problems with query view using its own controller<br />
*** Fixed module info not being displayed for query results<br />
*** Still no version info for query results<br />
*** Still an issue with version query view not being updated<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== Older meetings ==<br />
* [[Development/2015]]<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=13088Development2017-01-11T14:09:30Z<p>Remi: /* January 11, 2017 */</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2017 ==<br />
=== January 11, 2017 ===<br />
<br />
'''Updates'''<br />
* [RR] New VisTrails version in Debian [https://packages.qa.debian.org/v/vistrails.html]<br />
<br />
'''Items to Discuss'''<br />
* [RR] Collaboration with school of medicine to port HiC-bench to VisTrails<br />
** Have homemade workflow system based on R scripts and symlinks<br />
*** Each steps has multiple sets of parameters<br />
*** Submits PBS jobs for each step with each set of parameters, using PBS job metadata to wait on previous step<br />
*** Gets all results in directory structure <code>step3-parameters/step2-parameters/step1-parameters/files</code><br />
** VisTrails improves provenance capture and makes editing pipelines easier<br />
*** Need better support for aggregation (groupby operation might be good enough; pandas?)<br />
<br />
== 2016 ==<br />
=== December 14, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Single-instance code: over-engineered, but will work<br />
** (Have to avoid race conditions on startup, and OSes offer very different locking primitives)<br />
<br />
'''Items to Discuss'''<br />
* Email from Matt Dirks: combobox should work, ask for more code?<br />
* Email from marin.nl:<br />
** looks like they would benefit from new interpreter<br />
** how to store their consolidated provenance when they exchange .vt files around?<br />
<br />
=== November 30, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Working on single instance code for MARIN ([https://github.com/VisTrails/VisTrails/pull/1205 #1205])<br />
** Remote shutdown ready ([https://github.com/VisTrails/VisTrails/pull/1204 #1204])<br />
** Still improving it<br />
** Want to return something more structured to the client (success/fail code, stdout, stderr)<br />
** Should be usable without Qt<br />
* [DK] Shutting down VisTrails has issues: cleans up interpreter (to remove files) but also resets persistent pipeline for no reason<br />
* [DK] Working on batch execution using console_mode<br />
* [DK] Settings aliased parameters when using core.api?<br />
<br />
=== November 16, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [RR] Issue [https://github.com/VisTrails/VisTrails/issues/1200 #1200] from @samtux: need to set PyQt to API version 2 before ever importing it; using require_pyqt_api2() does that (also can install bundle)<br />
<br />
'''Items to Discuss'''<br />
<br />
* Email from Pieter: executing pipelines with parameters from command-line using single-instance<br />
** Seems to work, main instance gets full command-line and alters configuration temporarily<br />
** Not sure what his error is (VisTrails returns little information in a lot of cases); though execution error should return something more meaningful<br />
** Actions:<br />
*** Can add switch to kill main instance<br />
*** Can improve single-instance code to return more meaningful errors + machine-readable output from "client" VisTrails<br />
*** Add client/server code to <tt>core</tt> that doesn't use Qt? Also need a run.py-line entrypoint that doesn't use gui (calls console mode like <tt>gui.application.VistrailsApplicationSingleton#noninteractiveMode()</tt>)?<br />
<br />
=== November 2, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* New version 2.2-2 of Debian package, still [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=796994 wrong copyright], and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=748692 broken bundle installation]<br />
** Finally replied, says copyright issue will be fixed soon, working on including VisTrails 2.2.4 for Debian Stretch<br />
* Working on Docker package, low hanging fruit because we have CLTools, and useful for some workflow (also step towards CWL support)<br />
<br />
'''Items to Discuss'''<br />
<br />
=== October 5, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
<br />
* working toward a 3.0 release; [https://github.com/VisTrails/VisTrails/projects/1 project board]<br />
** unique ids<br />
** bundles<br />
** GUI fixes<br />
** wrapping updates<br />
** python3<br />
** interpreter?<br />
<br />
* command-line input parameters to provenance<br />
** [DK] we don't create a persisted workflow from those input params, i think<br />
** [DK] these are already defined as aliases so we treat them differently form a normal input param<br />
** [DK] still have the potential problem of allowing a user to view the workflow as it was run with a given set of aliases<br />
** Do the vistrails variables values get stored in provenance? (what if values change?)<br />
<br />
=== September 14, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] [http://aosabook.org/en/vistrails.html VisTrails] in [http://aosabook.org/ The Architecture of Open Source Applications] Vol I (2012)<br />
** Did we know/contribute to this?<br />
*** [DK] Yes. We wrote the text for the VisTrails chapter<br />
** Drafted response<br />
*** [DK] Looks good, will forward.<br />
<br />
=== August 31, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Went over vgc.poly.edu/vistrails.org config:<br />
** Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)<br />
** Fixed usersguide scripts (won't disappear again)<br />
** Moved some websites over to HTTPS<br />
** [https://docs.google.com/document/d/18YZewjR2YY9cQ9pzdA-nU9jbu2L7jYcPlDi3iaPadwA/edit Websites]<br />
* [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs ([http://vistrails.readthedocs.io/ currently there]) and users' guide under the same RTD "project"<br />
* [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that<br />
<br />
'''Items to Discuss'''<br />
* [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/<br />
** Get rid of more services? (svn, trac, ...)<br />
* [RR] Notebook execution works. How do we support more languages (than Python)?<br />
** Need: detection, metadata reader, code to input values<br />
* [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?<br />
<br />
=== August 17, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Fixed [https://github.com/VisTrails/VisTrails/commit/2cd42d156243067a7663bb0bf0156e5d87188c2f parameter order upgrade bug] that breaks VTK examples.<br />
* [TE] Matplotlib and sklearn are exportable using module-to-script api<br />
* [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.<br />
<br />
'''Items to Discuss'''<br />
* Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!<br />
* Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)<br />
* [TE] Wrap-up notes<br />
<br />
=== August 10, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Python Wrapper and scripting<br />
** All changes merged into python-wrapper branch<br />
*** Tests pass<br />
*** Still need to update sub-branches<br />
** Minor fixes<br />
*** vtkPlot3DReader upgrade bug<br />
*** Added tuple exporting<br />
** Most of vtk examples can be exported [https://gist.github.com/rexissimus/5ac7374cde8525d84f9443f019d234e2 example]<br />
*** Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary<br />
**** Could be a bug in the old vistrail, but not sure<br />
* [DK] Subworkflows and uuids<br />
<br />
=== August 3, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Wrapper patches now stored as [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/vtk_patch.py python code]<br />
** Connection variables now lowercase<br />
** Can we use code as output variables?<br />
*** E.g. "vtk_dataset_mapper.GetOutput(0)"<br />
*** Could mutate output when output method is only supposed to be called once<br />
*** Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)<br />
<br />
=== July 27, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] DB bundle serialization implemented<br />
<br />
'''Items to Discuss'''<br />
* [TE] Finished using patch in script [https://github.com/VisTrails/VisTrails/tree/python-wrapper-script-patches python-wrapper-script-patches]<br />
** locale?<br />
** [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d new patching example] vs [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 old patching example]<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/parse.py#L135-L273 Example of new patches] <br />
** Legacy '.vtkInstance' is now added as a patch "self.vtkInstance=self" (see new patching example)<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/api.py New api for executing standalone modules]<br />
*** Accessing module was disabled for core/api, but a different api for this is probably better anyway.<br />
*** api.Package now gives [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d#file-new_patching-py-L26 directly callable module compute functions]<br />
*** Only works on modules not using interpreter functionality ('.interpreter', etc.)<br />
**** for zip file modules and others that need temp files and use the interpreter for this, need to modify the module, but could work around with python libraries for managing temp files<br />
** Fixed more renaming issues - [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/scripts.py#L63-L64 function argument names should not be replaced]<br />
** Still need to do something nicer for the patch formatting<br />
*** patch keys currently contain non-python-variable characters but this can be fixed<br />
*** Need to replace string template with python variables that is replacable using RedBaron<br />
**** Patches could then look like:<br />
<br />
def vtkInstance():<br />
self.vtkInstance = self<br />
<br />
=== July 20, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] New patch method for python-wrapper [https://gist.github.com/rexissimus/009b303e4cadcd4b8f2fec6cf848cb2b example]<br />
** A patch is a string template with self/input/output variables for patching a class method <br />
** Patches are stored in xml specification<br />
** Patches are assigned to methods in ClassSpec's<br />
** Used both for execution in vistrails and when exporting as script<br />
*** Executed with "exec" when run in vistrails<br />
*** Turns patches into natural calls in script<br />
** Wrapping VTK library no longer needed<br />
*** Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)<br />
** Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)<br />
** Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)<br />
** TODO's<br />
*** Add to export_as_script<br />
*** Patch Modules that represent Python functions as well<br />
<br />
* [vistrails-users] Read data from SQL example?<br />
<br />
=== July 13, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] bundles on sql-alchemy branch<br />
** Bundle, BundleMapping, and BundleSerializer<br />
** New BaseSerializers to deal with versions of directory, zip, db serializers<br />
** Can register the BundleSerializers with the base serializers<br />
** Translation between bundle versions<br />
** Bundle.add_object "just works" if an appropriate mapping was registered<br />
<br />
'''Items to Discuss'''<br />
* [TE] Simplified running module using API for scripting [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 example]<br />
** API Module can be executed using '''compute''' method<br />
*** Takes list of outputs to compute + inputs as keyword arguments<br />
* Status of general cell synchronization (vistrails-users)<br />
<br />
=== July 6, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] Working on bundles<br />
<br />
'''Items to Discuss'''<br />
* [TE] Using vistrails API in scripting [https://gist.github.com/rexissimus/034d9c4dedc3299a079e84981dedd8c7 example 1 (API with VTK)] [https://gist.github.com/rexissimus/a4a9ca6c08d4058c64967d2b35415fd5 example 2 (DownloadFile only)]<br />
** Call module in script using vistrails API<br />
** Can be used when no export method is defined.<br />
* [TE] [https://github.com/VisTrails/VisTrails/commit/c76c109349769fee1280f85c22678380a2f9cd39 Server news bug] Critical?<br />
**[RR] No, unreleased<br />
* [DK] New bundle organization<br />
** BundleMapping contains individual mappings from raw objects to BundleObj<br />
** Must be separate in order to have in-memory, separate from serializer settings<br />
<br />
=== June 29, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as python script<br />
** Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) [https://gist.github.com/rexissimus/16caf5200802e150296f62a0c7844c5a example]<br />
** Preludes (imports) still need improvements<br />
** Works with vtkInteractionHandler (clipping plane)<br />
** Resulting clipping plane is incorrect? (supposed to hide one side of the plane)<br />
** vtkRendererOutput Uses VTKRenderWindow as output<br />
*** Should support different output modes?<br />
*** Does not work with multiple outputs (Next cell is displayed when current is closed)<br />
*** Uses wrapped vtkInstance (as vvtk)<br />
**** Create new patch system with code that can be used both in vistrails and in a python script?<br />
<br />
=== June 22, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fix for upgraded analogies (PR [https://github.com/VisTrails/VisTrails/pull/1177 #1177])<br />
** Uses module location to find module remap (The best we can do right now?)<br />
** Also fixes remapped port names for connections to existing modules <br />
** May still fail on complex upgrades like adding a function on an upgraded module that has a different port name<br />
<br />
* [TE] Working on exporting wrapped libraries (VTK)<br />
<br />
=== June 15, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Added depth logic to python script exporter ([https://github.com/VisTrails/VisTrails/commit/50f220e4f6abe5a7bb8689f73f456f43df2dbc12 full commit])<br />
** Fully implements connection merging and list looping<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Making analogy work on upgraded workflows ([https://github.com/VisTrails/VisTrails/issues/1175 #1175])<br />
** Analogy only works We need to use the non-upgraded action chain<br />
** But pipelines need to be valid (matcher uses module portspecs)<br />
** Tried to get upgrade remap from upgrade action, but this seems impossible<br />
** Use module location instead?<br />
*** Location should be identical for one-to-one module upgrades<br />
<br />
=== June 1, 2016 ===<br />
<br />
'''Updates'''<br />
* VisTrails 2.2.4 announced (finally)<br />
** 5 survey replies<br />
** 47 downloads last week<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as script (python-wrapper)<br />
** to_python_script<br />
*** needs the pipeline (or at least the connections)<br />
**** Should not generate code for any possible VTK method (exporter detects which ports is used in the source)<br />
**** VTK needs to know connection order<br />
**** Adding functions directly to the script results in nicer code<br />
** Add support for multiple input connections<br />
** Add support for list looping<br />
<br />
=== May 25, 2016 ===<br />
<br />
'''Updates'''<br />
* Mailing-list fixed<br />
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])<br />
<br />
'''Items to Discuss'''<br />
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)<br />
* [TE] Union ports<br />
** Show union ports in pipeline (DEMO)<br />
<br />
=== May 18, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] mailing list still down - Julio working on it<br />
* [TE] Improved union port<br />
** New schema 1.0.5 adds portitem.union string attribute<br />
** Works with pipeline view, parameter exploration view, and mashup view<br />
** Should work well for matplotlib<br />
** Use with VTK ports?<br />
*** It uses [item_1,item_2] style ports usually with different tuple length.<br />
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]<br />
<br />
=== May 11, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?<br />
*** Shows single port for all ports with same sort_key<br />
*** Select specific type when adding/showing<br />
*** Only one type should be visible on the module at any time<br />
*** Making tuple parameters union types is probably too complicated<br />
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly) <br />
<br />
* [TE] Feature suggestion: Let connections connect to functions/parameters<br />
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters<br />
** Use separate views for ports and functions (like before)<br />
*** Allow rearranging of functions<br />
** Pros:<br />
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.<br />
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.<br />
**** Makes List module simpler<br />
*** Connect to parameter in tuple<br />
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds. <br />
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.<br />
** Cons:<br />
*** Interface more complicated and crowded<br />
*** More difficult for users to grasp?<br />
*** No immediate use case<br />
*** Makes no union type for tuple parameters a bigger problem.<br />
<br />
* [TE] Publish user survey?<br />
** No response from vistrails release yet<br />
** Not announced on vistrails-users yet.<br />
<br />
=== May 4, 2016 ===<br />
'''Updates'''<br />
* VisTrails 2.2.4 released<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
<br />
=== April 27, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Wrapping Bokeh<br />
** New method type "operation"<br />
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output<br />
**** Glyph sets function name and arguments as output<br />
***** Not serializable, but should always be computed together with the subject of the operation.<br />
**** Figure applies Glyph operation on itself<br />
*** Useful for matplotlib?<br />
**** matplotlib currently find actors by searching upstream?<br />
**** But this is more difficult since matplotlib is not function-based.<br />
** New "union" port type<br />
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value<br />
**** Uses only first value in tuple<br />
**** Bokeh have many properties that support (list, string, Float)<br />
*** Does not work with depth=1 types (e.g. Float of depth 1)<br />
**** Turn Depth 1 types with known type into lists?<br />
<br />
* [TE] New bugfix release<br />
** Finalize survey<br />
<br />
=== April 20, 2016 ===<br />
'''Updates'''<br />
* Merged upgrade fixes<br />
** Now has infinite loop test<br />
** loop test now correctly loops 5 times<br />
*** remap_module used wrong (current) package version when creating ModuleDescriptor<br />
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])<br />
<br />
'''Items to Discuss'''<br />
* New bugfix release<br />
** Merge usagestats<br />
** Need to finalize survey<br />
<br />
=== April 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?<br />
<br />
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])<br />
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]<br />
** Cherry-pick test to preserve source?<br />
** Need maxPipelineFixAttempts option?<br />
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.<br />
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?<br />
*** Would need to be moved into handle_invalid_pipeline<br />
<br />
=== April 6, 2016 ===<br />
'''Updates'''<br />
* [TE] Query View improvements<br />
** Done except for displaying "Version Info" and #1164<br />
* [RR] Working on finishing new interpreter (finally!)<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Try latest upgrade first, then previous.<br />
** If all fail, select latest upgrade?<br />
<br />
=== March 30, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Query View improvements<br />
** Unified do_version_switch and get_upgrades as validate_version()<br />
** Problems with query view using its own controller<br />
*** Fixed module info not being displayed for query results<br />
*** Still no version info for query results<br />
*** Still an issue with version query view not being updated<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== Older meetings ==<br />
* [[Development/2015]]<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=12924Development2016-12-19T16:13:01Z<p>Remi: /* January 4, 2017 */</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2017 ==<br />
=== January 4, 2017 ===<br />
<br />
'''Updates'''<br />
* [RR] New VisTrails version in Debian [https://packages.qa.debian.org/v/vistrails.html]<br />
<br />
'''Items to Discuss'''<br />
<br />
== 2016 ==<br />
=== December 14, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Single-instance code: over-engineered, but will work<br />
** (Have to avoid race conditions on startup, and OSes offer very different locking primitives)<br />
<br />
'''Items to Discuss'''<br />
* Email from Matt Dirks: combobox should work, ask for more code?<br />
* Email from marin.nl:<br />
** looks like they would benefit from new interpreter<br />
** how to store their consolidated provenance when they exchange .vt files around?<br />
<br />
=== November 30, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Working on single instance code for MARIN ([https://github.com/VisTrails/VisTrails/pull/1205 #1205])<br />
** Remote shutdown ready ([https://github.com/VisTrails/VisTrails/pull/1204 #1204])<br />
** Still improving it<br />
** Want to return something more structured to the client (success/fail code, stdout, stderr)<br />
** Should be usable without Qt<br />
* [DK] Shutting down VisTrails has issues: cleans up interpreter (to remove files) but also resets persistent pipeline for no reason<br />
* [DK] Working on batch execution using console_mode<br />
* [DK] Settings aliased parameters when using core.api?<br />
<br />
=== November 16, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [RR] Issue [https://github.com/VisTrails/VisTrails/issues/1200 #1200] from @samtux: need to set PyQt to API version 2 before ever importing it; using require_pyqt_api2() does that (also can install bundle)<br />
<br />
'''Items to Discuss'''<br />
<br />
* Email from Pieter: executing pipelines with parameters from command-line using single-instance<br />
** Seems to work, main instance gets full command-line and alters configuration temporarily<br />
** Not sure what his error is (VisTrails returns little information in a lot of cases); though execution error should return something more meaningful<br />
** Actions:<br />
*** Can add switch to kill main instance<br />
*** Can improve single-instance code to return more meaningful errors + machine-readable output from "client" VisTrails<br />
*** Add client/server code to <tt>core</tt> that doesn't use Qt? Also need a run.py-line entrypoint that doesn't use gui (calls console mode like <tt>gui.application.VistrailsApplicationSingleton#noninteractiveMode()</tt>)?<br />
<br />
=== November 2, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* New version 2.2-2 of Debian package, still [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=796994 wrong copyright], and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=748692 broken bundle installation]<br />
** Finally replied, says copyright issue will be fixed soon, working on including VisTrails 2.2.4 for Debian Stretch<br />
* Working on Docker package, low hanging fruit because we have CLTools, and useful for some workflow (also step towards CWL support)<br />
<br />
'''Items to Discuss'''<br />
<br />
=== October 5, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
<br />
* working toward a 3.0 release; [https://github.com/VisTrails/VisTrails/projects/1 project board]<br />
** unique ids<br />
** bundles<br />
** GUI fixes<br />
** wrapping updates<br />
** python3<br />
** interpreter?<br />
<br />
* command-line input parameters to provenance<br />
** [DK] we don't create a persisted workflow from those input params, i think<br />
** [DK] these are already defined as aliases so we treat them differently form a normal input param<br />
** [DK] still have the potential problem of allowing a user to view the workflow as it was run with a given set of aliases<br />
** Do the vistrails variables values get stored in provenance? (what if values change?)<br />
<br />
=== September 14, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] [http://aosabook.org/en/vistrails.html VisTrails] in [http://aosabook.org/ The Architecture of Open Source Applications] Vol I (2012)<br />
** Did we know/contribute to this?<br />
*** [DK] Yes. We wrote the text for the VisTrails chapter<br />
** Drafted response<br />
*** [DK] Looks good, will forward.<br />
<br />
=== August 31, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Went over vgc.poly.edu/vistrails.org config:<br />
** Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)<br />
** Fixed usersguide scripts (won't disappear again)<br />
** Moved some websites over to HTTPS<br />
** [https://docs.google.com/document/d/18YZewjR2YY9cQ9pzdA-nU9jbu2L7jYcPlDi3iaPadwA/edit Websites]<br />
* [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs ([http://vistrails.readthedocs.io/ currently there]) and users' guide under the same RTD "project"<br />
* [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that<br />
<br />
'''Items to Discuss'''<br />
* [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/<br />
** Get rid of more services? (svn, trac, ...)<br />
* [RR] Notebook execution works. How do we support more languages (than Python)?<br />
** Need: detection, metadata reader, code to input values<br />
* [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?<br />
<br />
=== August 17, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Fixed [https://github.com/VisTrails/VisTrails/commit/2cd42d156243067a7663bb0bf0156e5d87188c2f parameter order upgrade bug] that breaks VTK examples.<br />
* [TE] Matplotlib and sklearn are exportable using module-to-script api<br />
* [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.<br />
<br />
'''Items to Discuss'''<br />
* Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!<br />
* Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)<br />
* [TE] Wrap-up notes<br />
<br />
=== August 10, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Python Wrapper and scripting<br />
** All changes merged into python-wrapper branch<br />
*** Tests pass<br />
*** Still need to update sub-branches<br />
** Minor fixes<br />
*** vtkPlot3DReader upgrade bug<br />
*** Added tuple exporting<br />
** Most of vtk examples can be exported [https://gist.github.com/rexissimus/5ac7374cde8525d84f9443f019d234e2 example]<br />
*** Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary<br />
**** Could be a bug in the old vistrail, but not sure<br />
* [DK] Subworkflows and uuids<br />
<br />
=== August 3, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Wrapper patches now stored as [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/vtk_patch.py python code]<br />
** Connection variables now lowercase<br />
** Can we use code as output variables?<br />
*** E.g. "vtk_dataset_mapper.GetOutput(0)"<br />
*** Could mutate output when output method is only supposed to be called once<br />
*** Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)<br />
<br />
=== July 27, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] DB bundle serialization implemented<br />
<br />
'''Items to Discuss'''<br />
* [TE] Finished using patch in script [https://github.com/VisTrails/VisTrails/tree/python-wrapper-script-patches python-wrapper-script-patches]<br />
** locale?<br />
** [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d new patching example] vs [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 old patching example]<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/parse.py#L135-L273 Example of new patches] <br />
** Legacy '.vtkInstance' is now added as a patch "self.vtkInstance=self" (see new patching example)<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/api.py New api for executing standalone modules]<br />
*** Accessing module was disabled for core/api, but a different api for this is probably better anyway.<br />
*** api.Package now gives [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d#file-new_patching-py-L26 directly callable module compute functions]<br />
*** Only works on modules not using interpreter functionality ('.interpreter', etc.)<br />
**** for zip file modules and others that need temp files and use the interpreter for this, need to modify the module, but could work around with python libraries for managing temp files<br />
** Fixed more renaming issues - [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/scripts.py#L63-L64 function argument names should not be replaced]<br />
** Still need to do something nicer for the patch formatting<br />
*** patch keys currently contain non-python-variable characters but this can be fixed<br />
*** Need to replace string template with python variables that is replacable using RedBaron<br />
**** Patches could then look like:<br />
<br />
def vtkInstance():<br />
self.vtkInstance = self<br />
<br />
=== July 20, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] New patch method for python-wrapper [https://gist.github.com/rexissimus/009b303e4cadcd4b8f2fec6cf848cb2b example]<br />
** A patch is a string template with self/input/output variables for patching a class method <br />
** Patches are stored in xml specification<br />
** Patches are assigned to methods in ClassSpec's<br />
** Used both for execution in vistrails and when exporting as script<br />
*** Executed with "exec" when run in vistrails<br />
*** Turns patches into natural calls in script<br />
** Wrapping VTK library no longer needed<br />
*** Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)<br />
** Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)<br />
** Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)<br />
** TODO's<br />
*** Add to export_as_script<br />
*** Patch Modules that represent Python functions as well<br />
<br />
* [vistrails-users] Read data from SQL example?<br />
<br />
=== July 13, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] bundles on sql-alchemy branch<br />
** Bundle, BundleMapping, and BundleSerializer<br />
** New BaseSerializers to deal with versions of directory, zip, db serializers<br />
** Can register the BundleSerializers with the base serializers<br />
** Translation between bundle versions<br />
** Bundle.add_object "just works" if an appropriate mapping was registered<br />
<br />
'''Items to Discuss'''<br />
* [TE] Simplified running module using API for scripting [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 example]<br />
** API Module can be executed using '''compute''' method<br />
*** Takes list of outputs to compute + inputs as keyword arguments<br />
* Status of general cell synchronization (vistrails-users)<br />
<br />
=== July 6, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] Working on bundles<br />
<br />
'''Items to Discuss'''<br />
* [TE] Using vistrails API in scripting [https://gist.github.com/rexissimus/034d9c4dedc3299a079e84981dedd8c7 example 1 (API with VTK)] [https://gist.github.com/rexissimus/a4a9ca6c08d4058c64967d2b35415fd5 example 2 (DownloadFile only)]<br />
** Call module in script using vistrails API<br />
** Can be used when no export method is defined.<br />
* [TE] [https://github.com/VisTrails/VisTrails/commit/c76c109349769fee1280f85c22678380a2f9cd39 Server news bug] Critical?<br />
**[RR] No, unreleased<br />
* [DK] New bundle organization<br />
** BundleMapping contains individual mappings from raw objects to BundleObj<br />
** Must be separate in order to have in-memory, separate from serializer settings<br />
<br />
=== June 29, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as python script<br />
** Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) [https://gist.github.com/rexissimus/16caf5200802e150296f62a0c7844c5a example]<br />
** Preludes (imports) still need improvements<br />
** Works with vtkInteractionHandler (clipping plane)<br />
** Resulting clipping plane is incorrect? (supposed to hide one side of the plane)<br />
** vtkRendererOutput Uses VTKRenderWindow as output<br />
*** Should support different output modes?<br />
*** Does not work with multiple outputs (Next cell is displayed when current is closed)<br />
*** Uses wrapped vtkInstance (as vvtk)<br />
**** Create new patch system with code that can be used both in vistrails and in a python script?<br />
<br />
=== June 22, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fix for upgraded analogies (PR [https://github.com/VisTrails/VisTrails/pull/1177 #1177])<br />
** Uses module location to find module remap (The best we can do right now?)<br />
** Also fixes remapped port names for connections to existing modules <br />
** May still fail on complex upgrades like adding a function on an upgraded module that has a different port name<br />
<br />
* [TE] Working on exporting wrapped libraries (VTK)<br />
<br />
=== June 15, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Added depth logic to python script exporter ([https://github.com/VisTrails/VisTrails/commit/50f220e4f6abe5a7bb8689f73f456f43df2dbc12 full commit])<br />
** Fully implements connection merging and list looping<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Making analogy work on upgraded workflows ([https://github.com/VisTrails/VisTrails/issues/1175 #1175])<br />
** Analogy only works We need to use the non-upgraded action chain<br />
** But pipelines need to be valid (matcher uses module portspecs)<br />
** Tried to get upgrade remap from upgrade action, but this seems impossible<br />
** Use module location instead?<br />
*** Location should be identical for one-to-one module upgrades<br />
<br />
=== June 1, 2016 ===<br />
<br />
'''Updates'''<br />
* VisTrails 2.2.4 announced (finally)<br />
** 5 survey replies<br />
** 47 downloads last week<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as script (python-wrapper)<br />
** to_python_script<br />
*** needs the pipeline (or at least the connections)<br />
**** Should not generate code for any possible VTK method (exporter detects which ports is used in the source)<br />
**** VTK needs to know connection order<br />
**** Adding functions directly to the script results in nicer code<br />
** Add support for multiple input connections<br />
** Add support for list looping<br />
<br />
=== May 25, 2016 ===<br />
<br />
'''Updates'''<br />
* Mailing-list fixed<br />
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])<br />
<br />
'''Items to Discuss'''<br />
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)<br />
* [TE] Union ports<br />
** Show union ports in pipeline (DEMO)<br />
<br />
=== May 18, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] mailing list still down - Julio working on it<br />
* [TE] Improved union port<br />
** New schema 1.0.5 adds portitem.union string attribute<br />
** Works with pipeline view, parameter exploration view, and mashup view<br />
** Should work well for matplotlib<br />
** Use with VTK ports?<br />
*** It uses [item_1,item_2] style ports usually with different tuple length.<br />
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]<br />
<br />
=== May 11, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?<br />
*** Shows single port for all ports with same sort_key<br />
*** Select specific type when adding/showing<br />
*** Only one type should be visible on the module at any time<br />
*** Making tuple parameters union types is probably too complicated<br />
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly) <br />
<br />
* [TE] Feature suggestion: Let connections connect to functions/parameters<br />
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters<br />
** Use separate views for ports and functions (like before)<br />
*** Allow rearranging of functions<br />
** Pros:<br />
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.<br />
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.<br />
**** Makes List module simpler<br />
*** Connect to parameter in tuple<br />
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds. <br />
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.<br />
** Cons:<br />
*** Interface more complicated and crowded<br />
*** More difficult for users to grasp?<br />
*** No immediate use case<br />
*** Makes no union type for tuple parameters a bigger problem.<br />
<br />
* [TE] Publish user survey?<br />
** No response from vistrails release yet<br />
** Not announced on vistrails-users yet.<br />
<br />
=== May 4, 2016 ===<br />
'''Updates'''<br />
* VisTrails 2.2.4 released<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
<br />
=== April 27, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Wrapping Bokeh<br />
** New method type "operation"<br />
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output<br />
**** Glyph sets function name and arguments as output<br />
***** Not serializable, but should always be computed together with the subject of the operation.<br />
**** Figure applies Glyph operation on itself<br />
*** Useful for matplotlib?<br />
**** matplotlib currently find actors by searching upstream?<br />
**** But this is more difficult since matplotlib is not function-based.<br />
** New "union" port type<br />
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value<br />
**** Uses only first value in tuple<br />
**** Bokeh have many properties that support (list, string, Float)<br />
*** Does not work with depth=1 types (e.g. Float of depth 1)<br />
**** Turn Depth 1 types with known type into lists?<br />
<br />
* [TE] New bugfix release<br />
** Finalize survey<br />
<br />
=== April 20, 2016 ===<br />
'''Updates'''<br />
* Merged upgrade fixes<br />
** Now has infinite loop test<br />
** loop test now correctly loops 5 times<br />
*** remap_module used wrong (current) package version when creating ModuleDescriptor<br />
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])<br />
<br />
'''Items to Discuss'''<br />
* New bugfix release<br />
** Merge usagestats<br />
** Need to finalize survey<br />
<br />
=== April 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?<br />
<br />
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])<br />
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]<br />
** Cherry-pick test to preserve source?<br />
** Need maxPipelineFixAttempts option?<br />
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.<br />
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?<br />
*** Would need to be moved into handle_invalid_pipeline<br />
<br />
=== April 6, 2016 ===<br />
'''Updates'''<br />
* [TE] Query View improvements<br />
** Done except for displaying "Version Info" and #1164<br />
* [RR] Working on finishing new interpreter (finally!)<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Try latest upgrade first, then previous.<br />
** If all fail, select latest upgrade?<br />
<br />
=== March 30, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Query View improvements<br />
** Unified do_version_switch and get_upgrades as validate_version()<br />
** Problems with query view using its own controller<br />
*** Fixed module info not being displayed for query results<br />
*** Still no version info for query results<br />
*** Still an issue with version query view not being updated<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== Older meetings ==<br />
* [[Development/2015]]<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=12871Development2016-12-14T16:09:08Z<p>Remi: /* December 14, 2016 */</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2016 ==<br />
=== December 14, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Single-instance code: over-engineered, but will work<br />
** (Have to avoid race conditions on startup, and OSes offer very different locking primitives)<br />
<br />
'''Items to Discuss'''<br />
* Email from Matt Dirks: combobox should work, ask for more code?<br />
* Email from marin.nl:<br />
** looks like they would benefit from new interpreter<br />
** how to store their consolidated provenance when they exchange .vt files around?<br />
<br />
=== November 30, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Working on single instance code for MARIN ([https://github.com/VisTrails/VisTrails/pull/1205 #1205])<br />
** Remote shutdown ready ([https://github.com/VisTrails/VisTrails/pull/1204 #1204])<br />
** Still improving it<br />
** Want to return something more structured to the client (success/fail code, stdout, stderr)<br />
** Should be usable without Qt<br />
* [DK] Shutting down VisTrails has issues: cleans up interpreter (to remove files) but also resets persistent pipeline for no reason<br />
* [DK] Working on batch execution using console_mode<br />
* [DK] Settings aliased parameters when using core.api?<br />
<br />
=== November 16, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [RR] Issue [https://github.com/VisTrails/VisTrails/issues/1200 #1200] from @samtux: need to set PyQt to API version 2 before ever importing it; using require_pyqt_api2() does that (also can install bundle)<br />
<br />
'''Items to Discuss'''<br />
<br />
* Email from Pieter: executing pipelines with parameters from command-line using single-instance<br />
** Seems to work, main instance gets full command-line and alters configuration temporarily<br />
** Not sure what his error is (VisTrails returns little information in a lot of cases); though execution error should return something more meaningful<br />
** Actions:<br />
*** Can add switch to kill main instance<br />
*** Can improve single-instance code to return more meaningful errors + machine-readable output from "client" VisTrails<br />
*** Add client/server code to <tt>core</tt> that doesn't use Qt? Also need a run.py-line entrypoint that doesn't use gui (calls console mode like <tt>gui.application.VistrailsApplicationSingleton#noninteractiveMode()</tt>)?<br />
<br />
=== November 2, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* New version 2.2-2 of Debian package, still [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=796994 wrong copyright], and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=748692 broken bundle installation]<br />
** Finally replied, says copyright issue will be fixed soon, working on including VisTrails 2.2.4 for Debian Stretch<br />
* Working on Docker package, low hanging fruit because we have CLTools, and useful for some workflow (also step towards CWL support)<br />
<br />
'''Items to Discuss'''<br />
<br />
=== October 5, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
<br />
* working toward a 3.0 release; [https://github.com/VisTrails/VisTrails/projects/1 project board]<br />
** unique ids<br />
** bundles<br />
** GUI fixes<br />
** wrapping updates<br />
** python3<br />
** interpreter?<br />
<br />
* command-line input parameters to provenance<br />
** [DK] we don't create a persisted workflow from those input params, i think<br />
** [DK] these are already defined as aliases so we treat them differently form a normal input param<br />
** [DK] still have the potential problem of allowing a user to view the workflow as it was run with a given set of aliases<br />
** Do the vistrails variables values get stored in provenance? (what if values change?)<br />
<br />
=== September 14, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] [http://aosabook.org/en/vistrails.html VisTrails] in [http://aosabook.org/ The Architecture of Open Source Applications] Vol I (2012)<br />
** Did we know/contribute to this?<br />
*** [DK] Yes. We wrote the text for the VisTrails chapter<br />
** Drafted response<br />
*** [DK] Looks good, will forward.<br />
<br />
=== August 31, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Went over vgc.poly.edu/vistrails.org config:<br />
** Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)<br />
** Fixed usersguide scripts (won't disappear again)<br />
** Moved some websites over to HTTPS<br />
** [https://docs.google.com/document/d/18YZewjR2YY9cQ9pzdA-nU9jbu2L7jYcPlDi3iaPadwA/edit Websites]<br />
* [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs ([http://vistrails.readthedocs.io/ currently there]) and users' guide under the same RTD "project"<br />
* [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that<br />
<br />
'''Items to Discuss'''<br />
* [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/<br />
** Get rid of more services? (svn, trac, ...)<br />
* [RR] Notebook execution works. How do we support more languages (than Python)?<br />
** Need: detection, metadata reader, code to input values<br />
* [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?<br />
<br />
=== August 17, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Fixed [https://github.com/VisTrails/VisTrails/commit/2cd42d156243067a7663bb0bf0156e5d87188c2f parameter order upgrade bug] that breaks VTK examples.<br />
* [TE] Matplotlib and sklearn are exportable using module-to-script api<br />
* [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.<br />
<br />
'''Items to Discuss'''<br />
* Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!<br />
* Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)<br />
* [TE] Wrap-up notes<br />
<br />
=== August 10, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Python Wrapper and scripting<br />
** All changes merged into python-wrapper branch<br />
*** Tests pass<br />
*** Still need to update sub-branches<br />
** Minor fixes<br />
*** vtkPlot3DReader upgrade bug<br />
*** Added tuple exporting<br />
** Most of vtk examples can be exported [https://gist.github.com/rexissimus/5ac7374cde8525d84f9443f019d234e2 example]<br />
*** Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary<br />
**** Could be a bug in the old vistrail, but not sure<br />
* [DK] Subworkflows and uuids<br />
<br />
=== August 3, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Wrapper patches now stored as [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/vtk_patch.py python code]<br />
** Connection variables now lowercase<br />
** Can we use code as output variables?<br />
*** E.g. "vtk_dataset_mapper.GetOutput(0)"<br />
*** Could mutate output when output method is only supposed to be called once<br />
*** Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)<br />
<br />
=== July 27, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] DB bundle serialization implemented<br />
<br />
'''Items to Discuss'''<br />
* [TE] Finished using patch in script [https://github.com/VisTrails/VisTrails/tree/python-wrapper-script-patches python-wrapper-script-patches]<br />
** locale?<br />
** [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d new patching example] vs [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 old patching example]<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/parse.py#L135-L273 Example of new patches] <br />
** Legacy '.vtkInstance' is now added as a patch "self.vtkInstance=self" (see new patching example)<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/api.py New api for executing standalone modules]<br />
*** Accessing module was disabled for core/api, but a different api for this is probably better anyway.<br />
*** api.Package now gives [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d#file-new_patching-py-L26 directly callable module compute functions]<br />
*** Only works on modules not using interpreter functionality ('.interpreter', etc.)<br />
**** for zip file modules and others that need temp files and use the interpreter for this, need to modify the module, but could work around with python libraries for managing temp files<br />
** Fixed more renaming issues - [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/scripts.py#L63-L64 function argument names should not be replaced]<br />
** Still need to do something nicer for the patch formatting<br />
*** patch keys currently contain non-python-variable characters but this can be fixed<br />
*** Need to replace string template with python variables that is replacable using RedBaron<br />
**** Patches could then look like:<br />
<br />
def vtkInstance():<br />
self.vtkInstance = self<br />
<br />
=== July 20, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] New patch method for python-wrapper [https://gist.github.com/rexissimus/009b303e4cadcd4b8f2fec6cf848cb2b example]<br />
** A patch is a string template with self/input/output variables for patching a class method <br />
** Patches are stored in xml specification<br />
** Patches are assigned to methods in ClassSpec's<br />
** Used both for execution in vistrails and when exporting as script<br />
*** Executed with "exec" when run in vistrails<br />
*** Turns patches into natural calls in script<br />
** Wrapping VTK library no longer needed<br />
*** Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)<br />
** Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)<br />
** Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)<br />
** TODO's<br />
*** Add to export_as_script<br />
*** Patch Modules that represent Python functions as well<br />
<br />
* [vistrails-users] Read data from SQL example?<br />
<br />
=== July 13, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] bundles on sql-alchemy branch<br />
** Bundle, BundleMapping, and BundleSerializer<br />
** New BaseSerializers to deal with versions of directory, zip, db serializers<br />
** Can register the BundleSerializers with the base serializers<br />
** Translation between bundle versions<br />
** Bundle.add_object "just works" if an appropriate mapping was registered<br />
<br />
'''Items to Discuss'''<br />
* [TE] Simplified running module using API for scripting [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 example]<br />
** API Module can be executed using '''compute''' method<br />
*** Takes list of outputs to compute + inputs as keyword arguments<br />
* Status of general cell synchronization (vistrails-users)<br />
<br />
=== July 6, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] Working on bundles<br />
<br />
'''Items to Discuss'''<br />
* [TE] Using vistrails API in scripting [https://gist.github.com/rexissimus/034d9c4dedc3299a079e84981dedd8c7 example 1 (API with VTK)] [https://gist.github.com/rexissimus/a4a9ca6c08d4058c64967d2b35415fd5 example 2 (DownloadFile only)]<br />
** Call module in script using vistrails API<br />
** Can be used when no export method is defined.<br />
* [TE] [https://github.com/VisTrails/VisTrails/commit/c76c109349769fee1280f85c22678380a2f9cd39 Server news bug] Critical?<br />
**[RR] No, unreleased<br />
* [DK] New bundle organization<br />
** BundleMapping contains individual mappings from raw objects to BundleObj<br />
** Must be separate in order to have in-memory, separate from serializer settings<br />
<br />
=== June 29, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as python script<br />
** Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) [https://gist.github.com/rexissimus/16caf5200802e150296f62a0c7844c5a example]<br />
** Preludes (imports) still need improvements<br />
** Works with vtkInteractionHandler (clipping plane)<br />
** Resulting clipping plane is incorrect? (supposed to hide one side of the plane)<br />
** vtkRendererOutput Uses VTKRenderWindow as output<br />
*** Should support different output modes?<br />
*** Does not work with multiple outputs (Next cell is displayed when current is closed)<br />
*** Uses wrapped vtkInstance (as vvtk)<br />
**** Create new patch system with code that can be used both in vistrails and in a python script?<br />
<br />
=== June 22, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fix for upgraded analogies (PR [https://github.com/VisTrails/VisTrails/pull/1177 #1177])<br />
** Uses module location to find module remap (The best we can do right now?)<br />
** Also fixes remapped port names for connections to existing modules <br />
** May still fail on complex upgrades like adding a function on an upgraded module that has a different port name<br />
<br />
* [TE] Working on exporting wrapped libraries (VTK)<br />
<br />
=== June 15, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Added depth logic to python script exporter ([https://github.com/VisTrails/VisTrails/commit/50f220e4f6abe5a7bb8689f73f456f43df2dbc12 full commit])<br />
** Fully implements connection merging and list looping<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Making analogy work on upgraded workflows ([https://github.com/VisTrails/VisTrails/issues/1175 #1175])<br />
** Analogy only works We need to use the non-upgraded action chain<br />
** But pipelines need to be valid (matcher uses module portspecs)<br />
** Tried to get upgrade remap from upgrade action, but this seems impossible<br />
** Use module location instead?<br />
*** Location should be identical for one-to-one module upgrades<br />
<br />
=== June 1, 2016 ===<br />
<br />
'''Updates'''<br />
* VisTrails 2.2.4 announced (finally)<br />
** 5 survey replies<br />
** 47 downloads last week<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as script (python-wrapper)<br />
** to_python_script<br />
*** needs the pipeline (or at least the connections)<br />
**** Should not generate code for any possible VTK method (exporter detects which ports is used in the source)<br />
**** VTK needs to know connection order<br />
**** Adding functions directly to the script results in nicer code<br />
** Add support for multiple input connections<br />
** Add support for list looping<br />
<br />
=== May 25, 2016 ===<br />
<br />
'''Updates'''<br />
* Mailing-list fixed<br />
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])<br />
<br />
'''Items to Discuss'''<br />
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)<br />
* [TE] Union ports<br />
** Show union ports in pipeline (DEMO)<br />
<br />
=== May 18, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] mailing list still down - Julio working on it<br />
* [TE] Improved union port<br />
** New schema 1.0.5 adds portitem.union string attribute<br />
** Works with pipeline view, parameter exploration view, and mashup view<br />
** Should work well for matplotlib<br />
** Use with VTK ports?<br />
*** It uses [item_1,item_2] style ports usually with different tuple length.<br />
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]<br />
<br />
=== May 11, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?<br />
*** Shows single port for all ports with same sort_key<br />
*** Select specific type when adding/showing<br />
*** Only one type should be visible on the module at any time<br />
*** Making tuple parameters union types is probably too complicated<br />
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly) <br />
<br />
* [TE] Feature suggestion: Let connections connect to functions/parameters<br />
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters<br />
** Use separate views for ports and functions (like before)<br />
*** Allow rearranging of functions<br />
** Pros:<br />
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.<br />
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.<br />
**** Makes List module simpler<br />
*** Connect to parameter in tuple<br />
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds. <br />
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.<br />
** Cons:<br />
*** Interface more complicated and crowded<br />
*** More difficult for users to grasp?<br />
*** No immediate use case<br />
*** Makes no union type for tuple parameters a bigger problem.<br />
<br />
* [TE] Publish user survey?<br />
** No response from vistrails release yet<br />
** Not announced on vistrails-users yet.<br />
<br />
=== May 4, 2016 ===<br />
'''Updates'''<br />
* VisTrails 2.2.4 released<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
<br />
=== April 27, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Wrapping Bokeh<br />
** New method type "operation"<br />
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output<br />
**** Glyph sets function name and arguments as output<br />
***** Not serializable, but should always be computed together with the subject of the operation.<br />
**** Figure applies Glyph operation on itself<br />
*** Useful for matplotlib?<br />
**** matplotlib currently find actors by searching upstream?<br />
**** But this is more difficult since matplotlib is not function-based.<br />
** New "union" port type<br />
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value<br />
**** Uses only first value in tuple<br />
**** Bokeh have many properties that support (list, string, Float)<br />
*** Does not work with depth=1 types (e.g. Float of depth 1)<br />
**** Turn Depth 1 types with known type into lists?<br />
<br />
* [TE] New bugfix release<br />
** Finalize survey<br />
<br />
=== April 20, 2016 ===<br />
'''Updates'''<br />
* Merged upgrade fixes<br />
** Now has infinite loop test<br />
** loop test now correctly loops 5 times<br />
*** remap_module used wrong (current) package version when creating ModuleDescriptor<br />
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])<br />
<br />
'''Items to Discuss'''<br />
* New bugfix release<br />
** Merge usagestats<br />
** Need to finalize survey<br />
<br />
=== April 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?<br />
<br />
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])<br />
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]<br />
** Cherry-pick test to preserve source?<br />
** Need maxPipelineFixAttempts option?<br />
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.<br />
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?<br />
*** Would need to be moved into handle_invalid_pipeline<br />
<br />
=== April 6, 2016 ===<br />
'''Updates'''<br />
* [TE] Query View improvements<br />
** Done except for displaying "Version Info" and #1164<br />
* [RR] Working on finishing new interpreter (finally!)<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Try latest upgrade first, then previous.<br />
** If all fail, select latest upgrade?<br />
<br />
=== March 30, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Query View improvements<br />
** Unified do_version_switch and get_upgrades as validate_version()<br />
** Problems with query view using its own controller<br />
*** Fixed module info not being displayed for query results<br />
*** Still no version info for query results<br />
*** Still an issue with version query view not being updated<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== Older meetings ==<br />
* [[Development/2015]]<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=12870Development2016-12-14T16:08:53Z<p>Remi: /* December 14, 2016 */</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2016 ==<br />
=== December 14, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Single-instance code: over-engineered, but will work<br />
* (Have to avoid race conditions on startup, and OSes offer very different locking primitives)<br />
<br />
'''Items to Discuss'''<br />
* Email from Matt Dirks: combobox should work, ask for more code?<br />
* Email from marin.nl:<br />
* looks like they would benefit from new interpreter<br />
* how to store their consolidated provenance when they exchange .vt files around?<br />
<br />
=== November 30, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Working on single instance code for MARIN ([https://github.com/VisTrails/VisTrails/pull/1205 #1205])<br />
** Remote shutdown ready ([https://github.com/VisTrails/VisTrails/pull/1204 #1204])<br />
** Still improving it<br />
** Want to return something more structured to the client (success/fail code, stdout, stderr)<br />
** Should be usable without Qt<br />
* [DK] Shutting down VisTrails has issues: cleans up interpreter (to remove files) but also resets persistent pipeline for no reason<br />
* [DK] Working on batch execution using console_mode<br />
* [DK] Settings aliased parameters when using core.api?<br />
<br />
=== November 16, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [RR] Issue [https://github.com/VisTrails/VisTrails/issues/1200 #1200] from @samtux: need to set PyQt to API version 2 before ever importing it; using require_pyqt_api2() does that (also can install bundle)<br />
<br />
'''Items to Discuss'''<br />
<br />
* Email from Pieter: executing pipelines with parameters from command-line using single-instance<br />
** Seems to work, main instance gets full command-line and alters configuration temporarily<br />
** Not sure what his error is (VisTrails returns little information in a lot of cases); though execution error should return something more meaningful<br />
** Actions:<br />
*** Can add switch to kill main instance<br />
*** Can improve single-instance code to return more meaningful errors + machine-readable output from "client" VisTrails<br />
*** Add client/server code to <tt>core</tt> that doesn't use Qt? Also need a run.py-line entrypoint that doesn't use gui (calls console mode like <tt>gui.application.VistrailsApplicationSingleton#noninteractiveMode()</tt>)?<br />
<br />
=== November 2, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* New version 2.2-2 of Debian package, still [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=796994 wrong copyright], and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=748692 broken bundle installation]<br />
** Finally replied, says copyright issue will be fixed soon, working on including VisTrails 2.2.4 for Debian Stretch<br />
* Working on Docker package, low hanging fruit because we have CLTools, and useful for some workflow (also step towards CWL support)<br />
<br />
'''Items to Discuss'''<br />
<br />
=== October 5, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
<br />
* working toward a 3.0 release; [https://github.com/VisTrails/VisTrails/projects/1 project board]<br />
** unique ids<br />
** bundles<br />
** GUI fixes<br />
** wrapping updates<br />
** python3<br />
** interpreter?<br />
<br />
* command-line input parameters to provenance<br />
** [DK] we don't create a persisted workflow from those input params, i think<br />
** [DK] these are already defined as aliases so we treat them differently form a normal input param<br />
** [DK] still have the potential problem of allowing a user to view the workflow as it was run with a given set of aliases<br />
** Do the vistrails variables values get stored in provenance? (what if values change?)<br />
<br />
=== September 14, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] [http://aosabook.org/en/vistrails.html VisTrails] in [http://aosabook.org/ The Architecture of Open Source Applications] Vol I (2012)<br />
** Did we know/contribute to this?<br />
*** [DK] Yes. We wrote the text for the VisTrails chapter<br />
** Drafted response<br />
*** [DK] Looks good, will forward.<br />
<br />
=== August 31, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Went over vgc.poly.edu/vistrails.org config:<br />
** Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)<br />
** Fixed usersguide scripts (won't disappear again)<br />
** Moved some websites over to HTTPS<br />
** [https://docs.google.com/document/d/18YZewjR2YY9cQ9pzdA-nU9jbu2L7jYcPlDi3iaPadwA/edit Websites]<br />
* [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs ([http://vistrails.readthedocs.io/ currently there]) and users' guide under the same RTD "project"<br />
* [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that<br />
<br />
'''Items to Discuss'''<br />
* [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/<br />
** Get rid of more services? (svn, trac, ...)<br />
* [RR] Notebook execution works. How do we support more languages (than Python)?<br />
** Need: detection, metadata reader, code to input values<br />
* [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?<br />
<br />
=== August 17, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Fixed [https://github.com/VisTrails/VisTrails/commit/2cd42d156243067a7663bb0bf0156e5d87188c2f parameter order upgrade bug] that breaks VTK examples.<br />
* [TE] Matplotlib and sklearn are exportable using module-to-script api<br />
* [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.<br />
<br />
'''Items to Discuss'''<br />
* Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!<br />
* Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)<br />
* [TE] Wrap-up notes<br />
<br />
=== August 10, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Python Wrapper and scripting<br />
** All changes merged into python-wrapper branch<br />
*** Tests pass<br />
*** Still need to update sub-branches<br />
** Minor fixes<br />
*** vtkPlot3DReader upgrade bug<br />
*** Added tuple exporting<br />
** Most of vtk examples can be exported [https://gist.github.com/rexissimus/5ac7374cde8525d84f9443f019d234e2 example]<br />
*** Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary<br />
**** Could be a bug in the old vistrail, but not sure<br />
* [DK] Subworkflows and uuids<br />
<br />
=== August 3, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Wrapper patches now stored as [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/vtk_patch.py python code]<br />
** Connection variables now lowercase<br />
** Can we use code as output variables?<br />
*** E.g. "vtk_dataset_mapper.GetOutput(0)"<br />
*** Could mutate output when output method is only supposed to be called once<br />
*** Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)<br />
<br />
=== July 27, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] DB bundle serialization implemented<br />
<br />
'''Items to Discuss'''<br />
* [TE] Finished using patch in script [https://github.com/VisTrails/VisTrails/tree/python-wrapper-script-patches python-wrapper-script-patches]<br />
** locale?<br />
** [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d new patching example] vs [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 old patching example]<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/parse.py#L135-L273 Example of new patches] <br />
** Legacy '.vtkInstance' is now added as a patch "self.vtkInstance=self" (see new patching example)<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/api.py New api for executing standalone modules]<br />
*** Accessing module was disabled for core/api, but a different api for this is probably better anyway.<br />
*** api.Package now gives [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d#file-new_patching-py-L26 directly callable module compute functions]<br />
*** Only works on modules not using interpreter functionality ('.interpreter', etc.)<br />
**** for zip file modules and others that need temp files and use the interpreter for this, need to modify the module, but could work around with python libraries for managing temp files<br />
** Fixed more renaming issues - [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/scripts.py#L63-L64 function argument names should not be replaced]<br />
** Still need to do something nicer for the patch formatting<br />
*** patch keys currently contain non-python-variable characters but this can be fixed<br />
*** Need to replace string template with python variables that is replacable using RedBaron<br />
**** Patches could then look like:<br />
<br />
def vtkInstance():<br />
self.vtkInstance = self<br />
<br />
=== July 20, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] New patch method for python-wrapper [https://gist.github.com/rexissimus/009b303e4cadcd4b8f2fec6cf848cb2b example]<br />
** A patch is a string template with self/input/output variables for patching a class method <br />
** Patches are stored in xml specification<br />
** Patches are assigned to methods in ClassSpec's<br />
** Used both for execution in vistrails and when exporting as script<br />
*** Executed with "exec" when run in vistrails<br />
*** Turns patches into natural calls in script<br />
** Wrapping VTK library no longer needed<br />
*** Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)<br />
** Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)<br />
** Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)<br />
** TODO's<br />
*** Add to export_as_script<br />
*** Patch Modules that represent Python functions as well<br />
<br />
* [vistrails-users] Read data from SQL example?<br />
<br />
=== July 13, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] bundles on sql-alchemy branch<br />
** Bundle, BundleMapping, and BundleSerializer<br />
** New BaseSerializers to deal with versions of directory, zip, db serializers<br />
** Can register the BundleSerializers with the base serializers<br />
** Translation between bundle versions<br />
** Bundle.add_object "just works" if an appropriate mapping was registered<br />
<br />
'''Items to Discuss'''<br />
* [TE] Simplified running module using API for scripting [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 example]<br />
** API Module can be executed using '''compute''' method<br />
*** Takes list of outputs to compute + inputs as keyword arguments<br />
* Status of general cell synchronization (vistrails-users)<br />
<br />
=== July 6, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] Working on bundles<br />
<br />
'''Items to Discuss'''<br />
* [TE] Using vistrails API in scripting [https://gist.github.com/rexissimus/034d9c4dedc3299a079e84981dedd8c7 example 1 (API with VTK)] [https://gist.github.com/rexissimus/a4a9ca6c08d4058c64967d2b35415fd5 example 2 (DownloadFile only)]<br />
** Call module in script using vistrails API<br />
** Can be used when no export method is defined.<br />
* [TE] [https://github.com/VisTrails/VisTrails/commit/c76c109349769fee1280f85c22678380a2f9cd39 Server news bug] Critical?<br />
**[RR] No, unreleased<br />
* [DK] New bundle organization<br />
** BundleMapping contains individual mappings from raw objects to BundleObj<br />
** Must be separate in order to have in-memory, separate from serializer settings<br />
<br />
=== June 29, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as python script<br />
** Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) [https://gist.github.com/rexissimus/16caf5200802e150296f62a0c7844c5a example]<br />
** Preludes (imports) still need improvements<br />
** Works with vtkInteractionHandler (clipping plane)<br />
** Resulting clipping plane is incorrect? (supposed to hide one side of the plane)<br />
** vtkRendererOutput Uses VTKRenderWindow as output<br />
*** Should support different output modes?<br />
*** Does not work with multiple outputs (Next cell is displayed when current is closed)<br />
*** Uses wrapped vtkInstance (as vvtk)<br />
**** Create new patch system with code that can be used both in vistrails and in a python script?<br />
<br />
=== June 22, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fix for upgraded analogies (PR [https://github.com/VisTrails/VisTrails/pull/1177 #1177])<br />
** Uses module location to find module remap (The best we can do right now?)<br />
** Also fixes remapped port names for connections to existing modules <br />
** May still fail on complex upgrades like adding a function on an upgraded module that has a different port name<br />
<br />
* [TE] Working on exporting wrapped libraries (VTK)<br />
<br />
=== June 15, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Added depth logic to python script exporter ([https://github.com/VisTrails/VisTrails/commit/50f220e4f6abe5a7bb8689f73f456f43df2dbc12 full commit])<br />
** Fully implements connection merging and list looping<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Making analogy work on upgraded workflows ([https://github.com/VisTrails/VisTrails/issues/1175 #1175])<br />
** Analogy only works We need to use the non-upgraded action chain<br />
** But pipelines need to be valid (matcher uses module portspecs)<br />
** Tried to get upgrade remap from upgrade action, but this seems impossible<br />
** Use module location instead?<br />
*** Location should be identical for one-to-one module upgrades<br />
<br />
=== June 1, 2016 ===<br />
<br />
'''Updates'''<br />
* VisTrails 2.2.4 announced (finally)<br />
** 5 survey replies<br />
** 47 downloads last week<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as script (python-wrapper)<br />
** to_python_script<br />
*** needs the pipeline (or at least the connections)<br />
**** Should not generate code for any possible VTK method (exporter detects which ports is used in the source)<br />
**** VTK needs to know connection order<br />
**** Adding functions directly to the script results in nicer code<br />
** Add support for multiple input connections<br />
** Add support for list looping<br />
<br />
=== May 25, 2016 ===<br />
<br />
'''Updates'''<br />
* Mailing-list fixed<br />
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])<br />
<br />
'''Items to Discuss'''<br />
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)<br />
* [TE] Union ports<br />
** Show union ports in pipeline (DEMO)<br />
<br />
=== May 18, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] mailing list still down - Julio working on it<br />
* [TE] Improved union port<br />
** New schema 1.0.5 adds portitem.union string attribute<br />
** Works with pipeline view, parameter exploration view, and mashup view<br />
** Should work well for matplotlib<br />
** Use with VTK ports?<br />
*** It uses [item_1,item_2] style ports usually with different tuple length.<br />
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]<br />
<br />
=== May 11, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?<br />
*** Shows single port for all ports with same sort_key<br />
*** Select specific type when adding/showing<br />
*** Only one type should be visible on the module at any time<br />
*** Making tuple parameters union types is probably too complicated<br />
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly) <br />
<br />
* [TE] Feature suggestion: Let connections connect to functions/parameters<br />
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters<br />
** Use separate views for ports and functions (like before)<br />
*** Allow rearranging of functions<br />
** Pros:<br />
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.<br />
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.<br />
**** Makes List module simpler<br />
*** Connect to parameter in tuple<br />
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds. <br />
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.<br />
** Cons:<br />
*** Interface more complicated and crowded<br />
*** More difficult for users to grasp?<br />
*** No immediate use case<br />
*** Makes no union type for tuple parameters a bigger problem.<br />
<br />
* [TE] Publish user survey?<br />
** No response from vistrails release yet<br />
** Not announced on vistrails-users yet.<br />
<br />
=== May 4, 2016 ===<br />
'''Updates'''<br />
* VisTrails 2.2.4 released<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
<br />
=== April 27, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Wrapping Bokeh<br />
** New method type "operation"<br />
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output<br />
**** Glyph sets function name and arguments as output<br />
***** Not serializable, but should always be computed together with the subject of the operation.<br />
**** Figure applies Glyph operation on itself<br />
*** Useful for matplotlib?<br />
**** matplotlib currently find actors by searching upstream?<br />
**** But this is more difficult since matplotlib is not function-based.<br />
** New "union" port type<br />
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value<br />
**** Uses only first value in tuple<br />
**** Bokeh have many properties that support (list, string, Float)<br />
*** Does not work with depth=1 types (e.g. Float of depth 1)<br />
**** Turn Depth 1 types with known type into lists?<br />
<br />
* [TE] New bugfix release<br />
** Finalize survey<br />
<br />
=== April 20, 2016 ===<br />
'''Updates'''<br />
* Merged upgrade fixes<br />
** Now has infinite loop test<br />
** loop test now correctly loops 5 times<br />
*** remap_module used wrong (current) package version when creating ModuleDescriptor<br />
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])<br />
<br />
'''Items to Discuss'''<br />
* New bugfix release<br />
** Merge usagestats<br />
** Need to finalize survey<br />
<br />
=== April 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?<br />
<br />
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])<br />
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]<br />
** Cherry-pick test to preserve source?<br />
** Need maxPipelineFixAttempts option?<br />
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.<br />
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?<br />
*** Would need to be moved into handle_invalid_pipeline<br />
<br />
=== April 6, 2016 ===<br />
'''Updates'''<br />
* [TE] Query View improvements<br />
** Done except for displaying "Version Info" and #1164<br />
* [RR] Working on finishing new interpreter (finally!)<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Try latest upgrade first, then previous.<br />
** If all fail, select latest upgrade?<br />
<br />
=== March 30, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Query View improvements<br />
** Unified do_version_switch and get_upgrades as validate_version()<br />
** Problems with query view using its own controller<br />
*** Fixed module info not being displayed for query results<br />
*** Still no version info for query results<br />
*** Still an issue with version query view not being updated<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== Older meetings ==<br />
* [[Development/2015]]<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=12869Development2016-12-14T16:04:24Z<p>Remi: /* December 14, 2016 */</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2016 ==<br />
=== December 14, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
=== November 30, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Working on single instance code for MARIN ([https://github.com/VisTrails/VisTrails/pull/1205 #1205])<br />
** Remote shutdown ready ([https://github.com/VisTrails/VisTrails/pull/1204 #1204])<br />
** Still improving it<br />
** Want to return something more structured to the client (success/fail code, stdout, stderr)<br />
** Should be usable without Qt<br />
* [DK] Shutting down VisTrails has issues: cleans up interpreter (to remove files) but also resets persistent pipeline for no reason<br />
* [DK] Working on batch execution using console_mode<br />
* [DK] Settings aliased parameters when using core.api?<br />
<br />
=== November 16, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [RR] Issue [https://github.com/VisTrails/VisTrails/issues/1200 #1200] from @samtux: need to set PyQt to API version 2 before ever importing it; using require_pyqt_api2() does that (also can install bundle)<br />
<br />
'''Items to Discuss'''<br />
<br />
* Email from Pieter: executing pipelines with parameters from command-line using single-instance<br />
** Seems to work, main instance gets full command-line and alters configuration temporarily<br />
** Not sure what his error is (VisTrails returns little information in a lot of cases); though execution error should return something more meaningful<br />
** Actions:<br />
*** Can add switch to kill main instance<br />
*** Can improve single-instance code to return more meaningful errors + machine-readable output from "client" VisTrails<br />
*** Add client/server code to <tt>core</tt> that doesn't use Qt? Also need a run.py-line entrypoint that doesn't use gui (calls console mode like <tt>gui.application.VistrailsApplicationSingleton#noninteractiveMode()</tt>)?<br />
<br />
=== November 2, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* New version 2.2-2 of Debian package, still [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=796994 wrong copyright], and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=748692 broken bundle installation]<br />
** Finally replied, says copyright issue will be fixed soon, working on including VisTrails 2.2.4 for Debian Stretch<br />
* Working on Docker package, low hanging fruit because we have CLTools, and useful for some workflow (also step towards CWL support)<br />
<br />
'''Items to Discuss'''<br />
<br />
=== October 5, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
<br />
* working toward a 3.0 release; [https://github.com/VisTrails/VisTrails/projects/1 project board]<br />
** unique ids<br />
** bundles<br />
** GUI fixes<br />
** wrapping updates<br />
** python3<br />
** interpreter?<br />
<br />
* command-line input parameters to provenance<br />
** [DK] we don't create a persisted workflow from those input params, i think<br />
** [DK] these are already defined as aliases so we treat them differently form a normal input param<br />
** [DK] still have the potential problem of allowing a user to view the workflow as it was run with a given set of aliases<br />
** Do the vistrails variables values get stored in provenance? (what if values change?)<br />
<br />
=== September 14, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] [http://aosabook.org/en/vistrails.html VisTrails] in [http://aosabook.org/ The Architecture of Open Source Applications] Vol I (2012)<br />
** Did we know/contribute to this?<br />
*** [DK] Yes. We wrote the text for the VisTrails chapter<br />
** Drafted response<br />
*** [DK] Looks good, will forward.<br />
<br />
=== August 31, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Went over vgc.poly.edu/vistrails.org config:<br />
** Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)<br />
** Fixed usersguide scripts (won't disappear again)<br />
** Moved some websites over to HTTPS<br />
** [https://docs.google.com/document/d/18YZewjR2YY9cQ9pzdA-nU9jbu2L7jYcPlDi3iaPadwA/edit Websites]<br />
* [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs ([http://vistrails.readthedocs.io/ currently there]) and users' guide under the same RTD "project"<br />
* [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that<br />
<br />
'''Items to Discuss'''<br />
* [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/<br />
** Get rid of more services? (svn, trac, ...)<br />
* [RR] Notebook execution works. How do we support more languages (than Python)?<br />
** Need: detection, metadata reader, code to input values<br />
* [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?<br />
<br />
=== August 17, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Fixed [https://github.com/VisTrails/VisTrails/commit/2cd42d156243067a7663bb0bf0156e5d87188c2f parameter order upgrade bug] that breaks VTK examples.<br />
* [TE] Matplotlib and sklearn are exportable using module-to-script api<br />
* [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.<br />
<br />
'''Items to Discuss'''<br />
* Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!<br />
* Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)<br />
* [TE] Wrap-up notes<br />
<br />
=== August 10, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Python Wrapper and scripting<br />
** All changes merged into python-wrapper branch<br />
*** Tests pass<br />
*** Still need to update sub-branches<br />
** Minor fixes<br />
*** vtkPlot3DReader upgrade bug<br />
*** Added tuple exporting<br />
** Most of vtk examples can be exported [https://gist.github.com/rexissimus/5ac7374cde8525d84f9443f019d234e2 example]<br />
*** Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary<br />
**** Could be a bug in the old vistrail, but not sure<br />
* [DK] Subworkflows and uuids<br />
<br />
=== August 3, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Wrapper patches now stored as [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/vtk_patch.py python code]<br />
** Connection variables now lowercase<br />
** Can we use code as output variables?<br />
*** E.g. "vtk_dataset_mapper.GetOutput(0)"<br />
*** Could mutate output when output method is only supposed to be called once<br />
*** Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)<br />
<br />
=== July 27, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] DB bundle serialization implemented<br />
<br />
'''Items to Discuss'''<br />
* [TE] Finished using patch in script [https://github.com/VisTrails/VisTrails/tree/python-wrapper-script-patches python-wrapper-script-patches]<br />
** locale?<br />
** [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d new patching example] vs [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 old patching example]<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/parse.py#L135-L273 Example of new patches] <br />
** Legacy '.vtkInstance' is now added as a patch "self.vtkInstance=self" (see new patching example)<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/api.py New api for executing standalone modules]<br />
*** Accessing module was disabled for core/api, but a different api for this is probably better anyway.<br />
*** api.Package now gives [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d#file-new_patching-py-L26 directly callable module compute functions]<br />
*** Only works on modules not using interpreter functionality ('.interpreter', etc.)<br />
**** for zip file modules and others that need temp files and use the interpreter for this, need to modify the module, but could work around with python libraries for managing temp files<br />
** Fixed more renaming issues - [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/scripts.py#L63-L64 function argument names should not be replaced]<br />
** Still need to do something nicer for the patch formatting<br />
*** patch keys currently contain non-python-variable characters but this can be fixed<br />
*** Need to replace string template with python variables that is replacable using RedBaron<br />
**** Patches could then look like:<br />
<br />
def vtkInstance():<br />
self.vtkInstance = self<br />
<br />
=== July 20, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] New patch method for python-wrapper [https://gist.github.com/rexissimus/009b303e4cadcd4b8f2fec6cf848cb2b example]<br />
** A patch is a string template with self/input/output variables for patching a class method <br />
** Patches are stored in xml specification<br />
** Patches are assigned to methods in ClassSpec's<br />
** Used both for execution in vistrails and when exporting as script<br />
*** Executed with "exec" when run in vistrails<br />
*** Turns patches into natural calls in script<br />
** Wrapping VTK library no longer needed<br />
*** Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)<br />
** Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)<br />
** Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)<br />
** TODO's<br />
*** Add to export_as_script<br />
*** Patch Modules that represent Python functions as well<br />
<br />
* [vistrails-users] Read data from SQL example?<br />
<br />
=== July 13, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] bundles on sql-alchemy branch<br />
** Bundle, BundleMapping, and BundleSerializer<br />
** New BaseSerializers to deal with versions of directory, zip, db serializers<br />
** Can register the BundleSerializers with the base serializers<br />
** Translation between bundle versions<br />
** Bundle.add_object "just works" if an appropriate mapping was registered<br />
<br />
'''Items to Discuss'''<br />
* [TE] Simplified running module using API for scripting [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 example]<br />
** API Module can be executed using '''compute''' method<br />
*** Takes list of outputs to compute + inputs as keyword arguments<br />
* Status of general cell synchronization (vistrails-users)<br />
<br />
=== July 6, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] Working on bundles<br />
<br />
'''Items to Discuss'''<br />
* [TE] Using vistrails API in scripting [https://gist.github.com/rexissimus/034d9c4dedc3299a079e84981dedd8c7 example 1 (API with VTK)] [https://gist.github.com/rexissimus/a4a9ca6c08d4058c64967d2b35415fd5 example 2 (DownloadFile only)]<br />
** Call module in script using vistrails API<br />
** Can be used when no export method is defined.<br />
* [TE] [https://github.com/VisTrails/VisTrails/commit/c76c109349769fee1280f85c22678380a2f9cd39 Server news bug] Critical?<br />
**[RR] No, unreleased<br />
* [DK] New bundle organization<br />
** BundleMapping contains individual mappings from raw objects to BundleObj<br />
** Must be separate in order to have in-memory, separate from serializer settings<br />
<br />
=== June 29, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as python script<br />
** Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) [https://gist.github.com/rexissimus/16caf5200802e150296f62a0c7844c5a example]<br />
** Preludes (imports) still need improvements<br />
** Works with vtkInteractionHandler (clipping plane)<br />
** Resulting clipping plane is incorrect? (supposed to hide one side of the plane)<br />
** vtkRendererOutput Uses VTKRenderWindow as output<br />
*** Should support different output modes?<br />
*** Does not work with multiple outputs (Next cell is displayed when current is closed)<br />
*** Uses wrapped vtkInstance (as vvtk)<br />
**** Create new patch system with code that can be used both in vistrails and in a python script?<br />
<br />
=== June 22, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fix for upgraded analogies (PR [https://github.com/VisTrails/VisTrails/pull/1177 #1177])<br />
** Uses module location to find module remap (The best we can do right now?)<br />
** Also fixes remapped port names for connections to existing modules <br />
** May still fail on complex upgrades like adding a function on an upgraded module that has a different port name<br />
<br />
* [TE] Working on exporting wrapped libraries (VTK)<br />
<br />
=== June 15, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Added depth logic to python script exporter ([https://github.com/VisTrails/VisTrails/commit/50f220e4f6abe5a7bb8689f73f456f43df2dbc12 full commit])<br />
** Fully implements connection merging and list looping<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Making analogy work on upgraded workflows ([https://github.com/VisTrails/VisTrails/issues/1175 #1175])<br />
** Analogy only works We need to use the non-upgraded action chain<br />
** But pipelines need to be valid (matcher uses module portspecs)<br />
** Tried to get upgrade remap from upgrade action, but this seems impossible<br />
** Use module location instead?<br />
*** Location should be identical for one-to-one module upgrades<br />
<br />
=== June 1, 2016 ===<br />
<br />
'''Updates'''<br />
* VisTrails 2.2.4 announced (finally)<br />
** 5 survey replies<br />
** 47 downloads last week<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as script (python-wrapper)<br />
** to_python_script<br />
*** needs the pipeline (or at least the connections)<br />
**** Should not generate code for any possible VTK method (exporter detects which ports is used in the source)<br />
**** VTK needs to know connection order<br />
**** Adding functions directly to the script results in nicer code<br />
** Add support for multiple input connections<br />
** Add support for list looping<br />
<br />
=== May 25, 2016 ===<br />
<br />
'''Updates'''<br />
* Mailing-list fixed<br />
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])<br />
<br />
'''Items to Discuss'''<br />
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)<br />
* [TE] Union ports<br />
** Show union ports in pipeline (DEMO)<br />
<br />
=== May 18, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] mailing list still down - Julio working on it<br />
* [TE] Improved union port<br />
** New schema 1.0.5 adds portitem.union string attribute<br />
** Works with pipeline view, parameter exploration view, and mashup view<br />
** Should work well for matplotlib<br />
** Use with VTK ports?<br />
*** It uses [item_1,item_2] style ports usually with different tuple length.<br />
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]<br />
<br />
=== May 11, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?<br />
*** Shows single port for all ports with same sort_key<br />
*** Select specific type when adding/showing<br />
*** Only one type should be visible on the module at any time<br />
*** Making tuple parameters union types is probably too complicated<br />
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly) <br />
<br />
* [TE] Feature suggestion: Let connections connect to functions/parameters<br />
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters<br />
** Use separate views for ports and functions (like before)<br />
*** Allow rearranging of functions<br />
** Pros:<br />
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.<br />
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.<br />
**** Makes List module simpler<br />
*** Connect to parameter in tuple<br />
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds. <br />
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.<br />
** Cons:<br />
*** Interface more complicated and crowded<br />
*** More difficult for users to grasp?<br />
*** No immediate use case<br />
*** Makes no union type for tuple parameters a bigger problem.<br />
<br />
* [TE] Publish user survey?<br />
** No response from vistrails release yet<br />
** Not announced on vistrails-users yet.<br />
<br />
=== May 4, 2016 ===<br />
'''Updates'''<br />
* VisTrails 2.2.4 released<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
<br />
=== April 27, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Wrapping Bokeh<br />
** New method type "operation"<br />
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output<br />
**** Glyph sets function name and arguments as output<br />
***** Not serializable, but should always be computed together with the subject of the operation.<br />
**** Figure applies Glyph operation on itself<br />
*** Useful for matplotlib?<br />
**** matplotlib currently find actors by searching upstream?<br />
**** But this is more difficult since matplotlib is not function-based.<br />
** New "union" port type<br />
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value<br />
**** Uses only first value in tuple<br />
**** Bokeh have many properties that support (list, string, Float)<br />
*** Does not work with depth=1 types (e.g. Float of depth 1)<br />
**** Turn Depth 1 types with known type into lists?<br />
<br />
* [TE] New bugfix release<br />
** Finalize survey<br />
<br />
=== April 20, 2016 ===<br />
'''Updates'''<br />
* Merged upgrade fixes<br />
** Now has infinite loop test<br />
** loop test now correctly loops 5 times<br />
*** remap_module used wrong (current) package version when creating ModuleDescriptor<br />
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])<br />
<br />
'''Items to Discuss'''<br />
* New bugfix release<br />
** Merge usagestats<br />
** Need to finalize survey<br />
<br />
=== April 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?<br />
<br />
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])<br />
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]<br />
** Cherry-pick test to preserve source?<br />
** Need maxPipelineFixAttempts option?<br />
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.<br />
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?<br />
*** Would need to be moved into handle_invalid_pipeline<br />
<br />
=== April 6, 2016 ===<br />
'''Updates'''<br />
* [TE] Query View improvements<br />
** Done except for displaying "Version Info" and #1164<br />
* [RR] Working on finishing new interpreter (finally!)<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Try latest upgrade first, then previous.<br />
** If all fail, select latest upgrade?<br />
<br />
=== March 30, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Query View improvements<br />
** Unified do_version_switch and get_upgrades as validate_version()<br />
** Problems with query view using its own controller<br />
*** Fixed module info not being displayed for query results<br />
*** Still no version info for query results<br />
*** Still an issue with version query view not being updated<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== Older meetings ==<br />
* [[Development/2015]]<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=12803Development2016-11-30T16:37:39Z<p>Remi: /* November 30, 2016 */</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2016 ==<br />
=== November 30, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Working on single instance code for MARIN ([https://github.com/VisTrails/VisTrails/pull/1205 #1205])<br />
** Remote shutdown ready ([https://github.com/VisTrails/VisTrails/pull/1204 #1204])<br />
** Still improving it<br />
** Want to return something more structured to the client (success/fail code, stdout, stderr)<br />
** Should be usable without Qt<br />
* [DK] Shutting down VisTrails has issues: cleans up interpreter (to remove files) but also resets persistent pipeline for no reason<br />
* [DK] Working on batch execution using console_mode<br />
* [DK] Settings aliased parameters when using core.api?<br />
<br />
=== November 16, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [RR] Issue [https://github.com/VisTrails/VisTrails/issues/1200 #1200] from @samtux: need to set PyQt to API version 2 before ever importing it; using require_pyqt_api2() does that (also can install bundle)<br />
<br />
'''Items to Discuss'''<br />
<br />
* Email from Pieter: executing pipelines with parameters from command-line using single-instance<br />
** Seems to work, main instance gets full command-line and alters configuration temporarily<br />
** Not sure what his error is (VisTrails returns little information in a lot of cases); though execution error should return something more meaningful<br />
** Actions:<br />
*** Can add switch to kill main instance<br />
*** Can improve single-instance code to return more meaningful errors + machine-readable output from "client" VisTrails<br />
*** Add client/server code to <tt>core</tt> that doesn't use Qt? Also need a run.py-line entrypoint that doesn't use gui (calls console mode like <tt>gui.application.VistrailsApplicationSingleton#noninteractiveMode()</tt>)?<br />
<br />
=== November 2, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* New version 2.2-2 of Debian package, still [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=796994 wrong copyright], and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=748692 broken bundle installation]<br />
** Finally replied, says copyright issue will be fixed soon, working on including VisTrails 2.2.4 for Debian Stretch<br />
* Working on Docker package, low hanging fruit because we have CLTools, and useful for some workflow (also step towards CWL support)<br />
<br />
'''Items to Discuss'''<br />
<br />
=== October 5, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
<br />
* working toward a 3.0 release; [https://github.com/VisTrails/VisTrails/projects/1 project board]<br />
** unique ids<br />
** bundles<br />
** GUI fixes<br />
** wrapping updates<br />
** python3<br />
** interpreter?<br />
<br />
* command-line input parameters to provenance<br />
** [DK] we don't create a persisted workflow from those input params, i think<br />
** [DK] these are already defined as aliases so we treat them differently form a normal input param<br />
** [DK] still have the potential problem of allowing a user to view the workflow as it was run with a given set of aliases<br />
** Do the vistrails variables values get stored in provenance? (what if values change?)<br />
<br />
=== September 14, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] [http://aosabook.org/en/vistrails.html VisTrails] in [http://aosabook.org/ The Architecture of Open Source Applications] Vol I (2012)<br />
** Did we know/contribute to this?<br />
*** [DK] Yes. We wrote the text for the VisTrails chapter<br />
** Drafted response<br />
*** [DK] Looks good, will forward.<br />
<br />
=== August 31, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Went over vgc.poly.edu/vistrails.org config:<br />
** Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)<br />
** Fixed usersguide scripts (won't disappear again)<br />
** Moved some websites over to HTTPS<br />
** [https://docs.google.com/document/d/18YZewjR2YY9cQ9pzdA-nU9jbu2L7jYcPlDi3iaPadwA/edit Websites]<br />
* [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs ([http://vistrails.readthedocs.io/ currently there]) and users' guide under the same RTD "project"<br />
* [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that<br />
<br />
'''Items to Discuss'''<br />
* [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/<br />
** Get rid of more services? (svn, trac, ...)<br />
* [RR] Notebook execution works. How do we support more languages (than Python)?<br />
** Need: detection, metadata reader, code to input values<br />
* [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?<br />
<br />
=== August 17, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Fixed [https://github.com/VisTrails/VisTrails/commit/2cd42d156243067a7663bb0bf0156e5d87188c2f parameter order upgrade bug] that breaks VTK examples.<br />
* [TE] Matplotlib and sklearn are exportable using module-to-script api<br />
* [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.<br />
<br />
'''Items to Discuss'''<br />
* Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!<br />
* Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)<br />
* [TE] Wrap-up notes<br />
<br />
=== August 10, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Python Wrapper and scripting<br />
** All changes merged into python-wrapper branch<br />
*** Tests pass<br />
*** Still need to update sub-branches<br />
** Minor fixes<br />
*** vtkPlot3DReader upgrade bug<br />
*** Added tuple exporting<br />
** Most of vtk examples can be exported [https://gist.github.com/rexissimus/5ac7374cde8525d84f9443f019d234e2 example]<br />
*** Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary<br />
**** Could be a bug in the old vistrail, but not sure<br />
* [DK] Subworkflows and uuids<br />
<br />
=== August 3, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Wrapper patches now stored as [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/vtk_patch.py python code]<br />
** Connection variables now lowercase<br />
** Can we use code as output variables?<br />
*** E.g. "vtk_dataset_mapper.GetOutput(0)"<br />
*** Could mutate output when output method is only supposed to be called once<br />
*** Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)<br />
<br />
=== July 27, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] DB bundle serialization implemented<br />
<br />
'''Items to Discuss'''<br />
* [TE] Finished using patch in script [https://github.com/VisTrails/VisTrails/tree/python-wrapper-script-patches python-wrapper-script-patches]<br />
** locale?<br />
** [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d new patching example] vs [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 old patching example]<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/parse.py#L135-L273 Example of new patches] <br />
** Legacy '.vtkInstance' is now added as a patch "self.vtkInstance=self" (see new patching example)<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/api.py New api for executing standalone modules]<br />
*** Accessing module was disabled for core/api, but a different api for this is probably better anyway.<br />
*** api.Package now gives [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d#file-new_patching-py-L26 directly callable module compute functions]<br />
*** Only works on modules not using interpreter functionality ('.interpreter', etc.)<br />
**** for zip file modules and others that need temp files and use the interpreter for this, need to modify the module, but could work around with python libraries for managing temp files<br />
** Fixed more renaming issues - [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/scripts.py#L63-L64 function argument names should not be replaced]<br />
** Still need to do something nicer for the patch formatting<br />
*** patch keys currently contain non-python-variable characters but this can be fixed<br />
*** Need to replace string template with python variables that is replacable using RedBaron<br />
**** Patches could then look like:<br />
<br />
def vtkInstance():<br />
self.vtkInstance = self<br />
<br />
=== July 20, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] New patch method for python-wrapper [https://gist.github.com/rexissimus/009b303e4cadcd4b8f2fec6cf848cb2b example]<br />
** A patch is a string template with self/input/output variables for patching a class method <br />
** Patches are stored in xml specification<br />
** Patches are assigned to methods in ClassSpec's<br />
** Used both for execution in vistrails and when exporting as script<br />
*** Executed with "exec" when run in vistrails<br />
*** Turns patches into natural calls in script<br />
** Wrapping VTK library no longer needed<br />
*** Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)<br />
** Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)<br />
** Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)<br />
** TODO's<br />
*** Add to export_as_script<br />
*** Patch Modules that represent Python functions as well<br />
<br />
* [vistrails-users] Read data from SQL example?<br />
<br />
=== July 13, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] bundles on sql-alchemy branch<br />
** Bundle, BundleMapping, and BundleSerializer<br />
** New BaseSerializers to deal with versions of directory, zip, db serializers<br />
** Can register the BundleSerializers with the base serializers<br />
** Translation between bundle versions<br />
** Bundle.add_object "just works" if an appropriate mapping was registered<br />
<br />
'''Items to Discuss'''<br />
* [TE] Simplified running module using API for scripting [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 example]<br />
** API Module can be executed using '''compute''' method<br />
*** Takes list of outputs to compute + inputs as keyword arguments<br />
* Status of general cell synchronization (vistrails-users)<br />
<br />
=== July 6, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] Working on bundles<br />
<br />
'''Items to Discuss'''<br />
* [TE] Using vistrails API in scripting [https://gist.github.com/rexissimus/034d9c4dedc3299a079e84981dedd8c7 example 1 (API with VTK)] [https://gist.github.com/rexissimus/a4a9ca6c08d4058c64967d2b35415fd5 example 2 (DownloadFile only)]<br />
** Call module in script using vistrails API<br />
** Can be used when no export method is defined.<br />
* [TE] [https://github.com/VisTrails/VisTrails/commit/c76c109349769fee1280f85c22678380a2f9cd39 Server news bug] Critical?<br />
**[RR] No, unreleased<br />
* [DK] New bundle organization<br />
** BundleMapping contains individual mappings from raw objects to BundleObj<br />
** Must be separate in order to have in-memory, separate from serializer settings<br />
<br />
=== June 29, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as python script<br />
** Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) [https://gist.github.com/rexissimus/16caf5200802e150296f62a0c7844c5a example]<br />
** Preludes (imports) still need improvements<br />
** Works with vtkInteractionHandler (clipping plane)<br />
** Resulting clipping plane is incorrect? (supposed to hide one side of the plane)<br />
** vtkRendererOutput Uses VTKRenderWindow as output<br />
*** Should support different output modes?<br />
*** Does not work with multiple outputs (Next cell is displayed when current is closed)<br />
*** Uses wrapped vtkInstance (as vvtk)<br />
**** Create new patch system with code that can be used both in vistrails and in a python script?<br />
<br />
=== June 22, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fix for upgraded analogies (PR [https://github.com/VisTrails/VisTrails/pull/1177 #1177])<br />
** Uses module location to find module remap (The best we can do right now?)<br />
** Also fixes remapped port names for connections to existing modules <br />
** May still fail on complex upgrades like adding a function on an upgraded module that has a different port name<br />
<br />
* [TE] Working on exporting wrapped libraries (VTK)<br />
<br />
=== June 15, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Added depth logic to python script exporter ([https://github.com/VisTrails/VisTrails/commit/50f220e4f6abe5a7bb8689f73f456f43df2dbc12 full commit])<br />
** Fully implements connection merging and list looping<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Making analogy work on upgraded workflows ([https://github.com/VisTrails/VisTrails/issues/1175 #1175])<br />
** Analogy only works We need to use the non-upgraded action chain<br />
** But pipelines need to be valid (matcher uses module portspecs)<br />
** Tried to get upgrade remap from upgrade action, but this seems impossible<br />
** Use module location instead?<br />
*** Location should be identical for one-to-one module upgrades<br />
<br />
=== June 1, 2016 ===<br />
<br />
'''Updates'''<br />
* VisTrails 2.2.4 announced (finally)<br />
** 5 survey replies<br />
** 47 downloads last week<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as script (python-wrapper)<br />
** to_python_script<br />
*** needs the pipeline (or at least the connections)<br />
**** Should not generate code for any possible VTK method (exporter detects which ports is used in the source)<br />
**** VTK needs to know connection order<br />
**** Adding functions directly to the script results in nicer code<br />
** Add support for multiple input connections<br />
** Add support for list looping<br />
<br />
=== May 25, 2016 ===<br />
<br />
'''Updates'''<br />
* Mailing-list fixed<br />
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])<br />
<br />
'''Items to Discuss'''<br />
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)<br />
* [TE] Union ports<br />
** Show union ports in pipeline (DEMO)<br />
<br />
=== May 18, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] mailing list still down - Julio working on it<br />
* [TE] Improved union port<br />
** New schema 1.0.5 adds portitem.union string attribute<br />
** Works with pipeline view, parameter exploration view, and mashup view<br />
** Should work well for matplotlib<br />
** Use with VTK ports?<br />
*** It uses [item_1,item_2] style ports usually with different tuple length.<br />
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]<br />
<br />
=== May 11, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?<br />
*** Shows single port for all ports with same sort_key<br />
*** Select specific type when adding/showing<br />
*** Only one type should be visible on the module at any time<br />
*** Making tuple parameters union types is probably too complicated<br />
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly) <br />
<br />
* [TE] Feature suggestion: Let connections connect to functions/parameters<br />
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters<br />
** Use separate views for ports and functions (like before)<br />
*** Allow rearranging of functions<br />
** Pros:<br />
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.<br />
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.<br />
**** Makes List module simpler<br />
*** Connect to parameter in tuple<br />
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds. <br />
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.<br />
** Cons:<br />
*** Interface more complicated and crowded<br />
*** More difficult for users to grasp?<br />
*** No immediate use case<br />
*** Makes no union type for tuple parameters a bigger problem.<br />
<br />
* [TE] Publish user survey?<br />
** No response from vistrails release yet<br />
** Not announced on vistrails-users yet.<br />
<br />
=== May 4, 2016 ===<br />
'''Updates'''<br />
* VisTrails 2.2.4 released<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
<br />
=== April 27, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Wrapping Bokeh<br />
** New method type "operation"<br />
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output<br />
**** Glyph sets function name and arguments as output<br />
***** Not serializable, but should always be computed together with the subject of the operation.<br />
**** Figure applies Glyph operation on itself<br />
*** Useful for matplotlib?<br />
**** matplotlib currently find actors by searching upstream?<br />
**** But this is more difficult since matplotlib is not function-based.<br />
** New "union" port type<br />
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value<br />
**** Uses only first value in tuple<br />
**** Bokeh have many properties that support (list, string, Float)<br />
*** Does not work with depth=1 types (e.g. Float of depth 1)<br />
**** Turn Depth 1 types with known type into lists?<br />
<br />
* [TE] New bugfix release<br />
** Finalize survey<br />
<br />
=== April 20, 2016 ===<br />
'''Updates'''<br />
* Merged upgrade fixes<br />
** Now has infinite loop test<br />
** loop test now correctly loops 5 times<br />
*** remap_module used wrong (current) package version when creating ModuleDescriptor<br />
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])<br />
<br />
'''Items to Discuss'''<br />
* New bugfix release<br />
** Merge usagestats<br />
** Need to finalize survey<br />
<br />
=== April 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?<br />
<br />
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])<br />
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]<br />
** Cherry-pick test to preserve source?<br />
** Need maxPipelineFixAttempts option?<br />
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.<br />
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?<br />
*** Would need to be moved into handle_invalid_pipeline<br />
<br />
=== April 6, 2016 ===<br />
'''Updates'''<br />
* [TE] Query View improvements<br />
** Done except for displaying "Version Info" and #1164<br />
* [RR] Working on finishing new interpreter (finally!)<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Try latest upgrade first, then previous.<br />
** If all fail, select latest upgrade?<br />
<br />
=== March 30, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Query View improvements<br />
** Unified do_version_switch and get_upgrades as validate_version()<br />
** Problems with query view using its own controller<br />
*** Fixed module info not being displayed for query results<br />
*** Still no version info for query results<br />
*** Still an issue with version query view not being updated<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== Older meetings ==<br />
* [[Development/2015]]<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=12802Development2016-11-30T16:37:02Z<p>Remi: /* November 30, 2016 */</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2016 ==<br />
=== November 30, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Working on single instance code for MARIN ([https://github.com/VisTrails/VisTrails/pull/1205 #1205])<br />
** Remote shutdown ready ([https://github.com/VisTrails/VisTrails/pull/1204 #1204])<br />
** Still improving it<br />
** Want to return something more structured to the client (success/fail code, stdout, stderr)<br />
** Should be usable without Qt<br />
* [DK] Shutting down VisTrails has issues: cleans up interpreter (to remove files) but also resets persistent pipeline for no reason<br />
* [DK] Working on batch execution using console_mode<br />
<br />
=== November 16, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [RR] Issue [https://github.com/VisTrails/VisTrails/issues/1200 #1200] from @samtux: need to set PyQt to API version 2 before ever importing it; using require_pyqt_api2() does that (also can install bundle)<br />
<br />
'''Items to Discuss'''<br />
<br />
* Email from Pieter: executing pipelines with parameters from command-line using single-instance<br />
** Seems to work, main instance gets full command-line and alters configuration temporarily<br />
** Not sure what his error is (VisTrails returns little information in a lot of cases); though execution error should return something more meaningful<br />
** Actions:<br />
*** Can add switch to kill main instance<br />
*** Can improve single-instance code to return more meaningful errors + machine-readable output from "client" VisTrails<br />
*** Add client/server code to <tt>core</tt> that doesn't use Qt? Also need a run.py-line entrypoint that doesn't use gui (calls console mode like <tt>gui.application.VistrailsApplicationSingleton#noninteractiveMode()</tt>)?<br />
<br />
=== November 2, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* New version 2.2-2 of Debian package, still [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=796994 wrong copyright], and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=748692 broken bundle installation]<br />
** Finally replied, says copyright issue will be fixed soon, working on including VisTrails 2.2.4 for Debian Stretch<br />
* Working on Docker package, low hanging fruit because we have CLTools, and useful for some workflow (also step towards CWL support)<br />
<br />
'''Items to Discuss'''<br />
<br />
=== October 5, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
<br />
* working toward a 3.0 release; [https://github.com/VisTrails/VisTrails/projects/1 project board]<br />
** unique ids<br />
** bundles<br />
** GUI fixes<br />
** wrapping updates<br />
** python3<br />
** interpreter?<br />
<br />
* command-line input parameters to provenance<br />
** [DK] we don't create a persisted workflow from those input params, i think<br />
** [DK] these are already defined as aliases so we treat them differently form a normal input param<br />
** [DK] still have the potential problem of allowing a user to view the workflow as it was run with a given set of aliases<br />
** Do the vistrails variables values get stored in provenance? (what if values change?)<br />
<br />
=== September 14, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] [http://aosabook.org/en/vistrails.html VisTrails] in [http://aosabook.org/ The Architecture of Open Source Applications] Vol I (2012)<br />
** Did we know/contribute to this?<br />
*** [DK] Yes. We wrote the text for the VisTrails chapter<br />
** Drafted response<br />
*** [DK] Looks good, will forward.<br />
<br />
=== August 31, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Went over vgc.poly.edu/vistrails.org config:<br />
** Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)<br />
** Fixed usersguide scripts (won't disappear again)<br />
** Moved some websites over to HTTPS<br />
** [https://docs.google.com/document/d/18YZewjR2YY9cQ9pzdA-nU9jbu2L7jYcPlDi3iaPadwA/edit Websites]<br />
* [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs ([http://vistrails.readthedocs.io/ currently there]) and users' guide under the same RTD "project"<br />
* [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that<br />
<br />
'''Items to Discuss'''<br />
* [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/<br />
** Get rid of more services? (svn, trac, ...)<br />
* [RR] Notebook execution works. How do we support more languages (than Python)?<br />
** Need: detection, metadata reader, code to input values<br />
* [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?<br />
<br />
=== August 17, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Fixed [https://github.com/VisTrails/VisTrails/commit/2cd42d156243067a7663bb0bf0156e5d87188c2f parameter order upgrade bug] that breaks VTK examples.<br />
* [TE] Matplotlib and sklearn are exportable using module-to-script api<br />
* [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.<br />
<br />
'''Items to Discuss'''<br />
* Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!<br />
* Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)<br />
* [TE] Wrap-up notes<br />
<br />
=== August 10, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Python Wrapper and scripting<br />
** All changes merged into python-wrapper branch<br />
*** Tests pass<br />
*** Still need to update sub-branches<br />
** Minor fixes<br />
*** vtkPlot3DReader upgrade bug<br />
*** Added tuple exporting<br />
** Most of vtk examples can be exported [https://gist.github.com/rexissimus/5ac7374cde8525d84f9443f019d234e2 example]<br />
*** Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary<br />
**** Could be a bug in the old vistrail, but not sure<br />
* [DK] Subworkflows and uuids<br />
<br />
=== August 3, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Wrapper patches now stored as [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/vtk_patch.py python code]<br />
** Connection variables now lowercase<br />
** Can we use code as output variables?<br />
*** E.g. "vtk_dataset_mapper.GetOutput(0)"<br />
*** Could mutate output when output method is only supposed to be called once<br />
*** Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)<br />
<br />
=== July 27, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] DB bundle serialization implemented<br />
<br />
'''Items to Discuss'''<br />
* [TE] Finished using patch in script [https://github.com/VisTrails/VisTrails/tree/python-wrapper-script-patches python-wrapper-script-patches]<br />
** locale?<br />
** [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d new patching example] vs [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 old patching example]<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/parse.py#L135-L273 Example of new patches] <br />
** Legacy '.vtkInstance' is now added as a patch "self.vtkInstance=self" (see new patching example)<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/api.py New api for executing standalone modules]<br />
*** Accessing module was disabled for core/api, but a different api for this is probably better anyway.<br />
*** api.Package now gives [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d#file-new_patching-py-L26 directly callable module compute functions]<br />
*** Only works on modules not using interpreter functionality ('.interpreter', etc.)<br />
**** for zip file modules and others that need temp files and use the interpreter for this, need to modify the module, but could work around with python libraries for managing temp files<br />
** Fixed more renaming issues - [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/scripts.py#L63-L64 function argument names should not be replaced]<br />
** Still need to do something nicer for the patch formatting<br />
*** patch keys currently contain non-python-variable characters but this can be fixed<br />
*** Need to replace string template with python variables that is replacable using RedBaron<br />
**** Patches could then look like:<br />
<br />
def vtkInstance():<br />
self.vtkInstance = self<br />
<br />
=== July 20, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] New patch method for python-wrapper [https://gist.github.com/rexissimus/009b303e4cadcd4b8f2fec6cf848cb2b example]<br />
** A patch is a string template with self/input/output variables for patching a class method <br />
** Patches are stored in xml specification<br />
** Patches are assigned to methods in ClassSpec's<br />
** Used both for execution in vistrails and when exporting as script<br />
*** Executed with "exec" when run in vistrails<br />
*** Turns patches into natural calls in script<br />
** Wrapping VTK library no longer needed<br />
*** Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)<br />
** Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)<br />
** Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)<br />
** TODO's<br />
*** Add to export_as_script<br />
*** Patch Modules that represent Python functions as well<br />
<br />
* [vistrails-users] Read data from SQL example?<br />
<br />
=== July 13, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] bundles on sql-alchemy branch<br />
** Bundle, BundleMapping, and BundleSerializer<br />
** New BaseSerializers to deal with versions of directory, zip, db serializers<br />
** Can register the BundleSerializers with the base serializers<br />
** Translation between bundle versions<br />
** Bundle.add_object "just works" if an appropriate mapping was registered<br />
<br />
'''Items to Discuss'''<br />
* [TE] Simplified running module using API for scripting [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 example]<br />
** API Module can be executed using '''compute''' method<br />
*** Takes list of outputs to compute + inputs as keyword arguments<br />
* Status of general cell synchronization (vistrails-users)<br />
<br />
=== July 6, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] Working on bundles<br />
<br />
'''Items to Discuss'''<br />
* [TE] Using vistrails API in scripting [https://gist.github.com/rexissimus/034d9c4dedc3299a079e84981dedd8c7 example 1 (API with VTK)] [https://gist.github.com/rexissimus/a4a9ca6c08d4058c64967d2b35415fd5 example 2 (DownloadFile only)]<br />
** Call module in script using vistrails API<br />
** Can be used when no export method is defined.<br />
* [TE] [https://github.com/VisTrails/VisTrails/commit/c76c109349769fee1280f85c22678380a2f9cd39 Server news bug] Critical?<br />
**[RR] No, unreleased<br />
* [DK] New bundle organization<br />
** BundleMapping contains individual mappings from raw objects to BundleObj<br />
** Must be separate in order to have in-memory, separate from serializer settings<br />
<br />
=== June 29, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as python script<br />
** Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) [https://gist.github.com/rexissimus/16caf5200802e150296f62a0c7844c5a example]<br />
** Preludes (imports) still need improvements<br />
** Works with vtkInteractionHandler (clipping plane)<br />
** Resulting clipping plane is incorrect? (supposed to hide one side of the plane)<br />
** vtkRendererOutput Uses VTKRenderWindow as output<br />
*** Should support different output modes?<br />
*** Does not work with multiple outputs (Next cell is displayed when current is closed)<br />
*** Uses wrapped vtkInstance (as vvtk)<br />
**** Create new patch system with code that can be used both in vistrails and in a python script?<br />
<br />
=== June 22, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fix for upgraded analogies (PR [https://github.com/VisTrails/VisTrails/pull/1177 #1177])<br />
** Uses module location to find module remap (The best we can do right now?)<br />
** Also fixes remapped port names for connections to existing modules <br />
** May still fail on complex upgrades like adding a function on an upgraded module that has a different port name<br />
<br />
* [TE] Working on exporting wrapped libraries (VTK)<br />
<br />
=== June 15, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Added depth logic to python script exporter ([https://github.com/VisTrails/VisTrails/commit/50f220e4f6abe5a7bb8689f73f456f43df2dbc12 full commit])<br />
** Fully implements connection merging and list looping<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Making analogy work on upgraded workflows ([https://github.com/VisTrails/VisTrails/issues/1175 #1175])<br />
** Analogy only works We need to use the non-upgraded action chain<br />
** But pipelines need to be valid (matcher uses module portspecs)<br />
** Tried to get upgrade remap from upgrade action, but this seems impossible<br />
** Use module location instead?<br />
*** Location should be identical for one-to-one module upgrades<br />
<br />
=== June 1, 2016 ===<br />
<br />
'''Updates'''<br />
* VisTrails 2.2.4 announced (finally)<br />
** 5 survey replies<br />
** 47 downloads last week<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as script (python-wrapper)<br />
** to_python_script<br />
*** needs the pipeline (or at least the connections)<br />
**** Should not generate code for any possible VTK method (exporter detects which ports is used in the source)<br />
**** VTK needs to know connection order<br />
**** Adding functions directly to the script results in nicer code<br />
** Add support for multiple input connections<br />
** Add support for list looping<br />
<br />
=== May 25, 2016 ===<br />
<br />
'''Updates'''<br />
* Mailing-list fixed<br />
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])<br />
<br />
'''Items to Discuss'''<br />
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)<br />
* [TE] Union ports<br />
** Show union ports in pipeline (DEMO)<br />
<br />
=== May 18, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] mailing list still down - Julio working on it<br />
* [TE] Improved union port<br />
** New schema 1.0.5 adds portitem.union string attribute<br />
** Works with pipeline view, parameter exploration view, and mashup view<br />
** Should work well for matplotlib<br />
** Use with VTK ports?<br />
*** It uses [item_1,item_2] style ports usually with different tuple length.<br />
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]<br />
<br />
=== May 11, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?<br />
*** Shows single port for all ports with same sort_key<br />
*** Select specific type when adding/showing<br />
*** Only one type should be visible on the module at any time<br />
*** Making tuple parameters union types is probably too complicated<br />
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly) <br />
<br />
* [TE] Feature suggestion: Let connections connect to functions/parameters<br />
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters<br />
** Use separate views for ports and functions (like before)<br />
*** Allow rearranging of functions<br />
** Pros:<br />
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.<br />
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.<br />
**** Makes List module simpler<br />
*** Connect to parameter in tuple<br />
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds. <br />
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.<br />
** Cons:<br />
*** Interface more complicated and crowded<br />
*** More difficult for users to grasp?<br />
*** No immediate use case<br />
*** Makes no union type for tuple parameters a bigger problem.<br />
<br />
* [TE] Publish user survey?<br />
** No response from vistrails release yet<br />
** Not announced on vistrails-users yet.<br />
<br />
=== May 4, 2016 ===<br />
'''Updates'''<br />
* VisTrails 2.2.4 released<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
<br />
=== April 27, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Wrapping Bokeh<br />
** New method type "operation"<br />
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output<br />
**** Glyph sets function name and arguments as output<br />
***** Not serializable, but should always be computed together with the subject of the operation.<br />
**** Figure applies Glyph operation on itself<br />
*** Useful for matplotlib?<br />
**** matplotlib currently find actors by searching upstream?<br />
**** But this is more difficult since matplotlib is not function-based.<br />
** New "union" port type<br />
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value<br />
**** Uses only first value in tuple<br />
**** Bokeh have many properties that support (list, string, Float)<br />
*** Does not work with depth=1 types (e.g. Float of depth 1)<br />
**** Turn Depth 1 types with known type into lists?<br />
<br />
* [TE] New bugfix release<br />
** Finalize survey<br />
<br />
=== April 20, 2016 ===<br />
'''Updates'''<br />
* Merged upgrade fixes<br />
** Now has infinite loop test<br />
** loop test now correctly loops 5 times<br />
*** remap_module used wrong (current) package version when creating ModuleDescriptor<br />
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])<br />
<br />
'''Items to Discuss'''<br />
* New bugfix release<br />
** Merge usagestats<br />
** Need to finalize survey<br />
<br />
=== April 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?<br />
<br />
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])<br />
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]<br />
** Cherry-pick test to preserve source?<br />
** Need maxPipelineFixAttempts option?<br />
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.<br />
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?<br />
*** Would need to be moved into handle_invalid_pipeline<br />
<br />
=== April 6, 2016 ===<br />
'''Updates'''<br />
* [TE] Query View improvements<br />
** Done except for displaying "Version Info" and #1164<br />
* [RR] Working on finishing new interpreter (finally!)<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Try latest upgrade first, then previous.<br />
** If all fail, select latest upgrade?<br />
<br />
=== March 30, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Query View improvements<br />
** Unified do_version_switch and get_upgrades as validate_version()<br />
** Problems with query view using its own controller<br />
*** Fixed module info not being displayed for query results<br />
*** Still no version info for query results<br />
*** Still an issue with version query view not being updated<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== Older meetings ==<br />
* [[Development/2015]]<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=12801Development2016-11-30T16:13:59Z<p>Remi: /* November 30, 2016 */</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2016 ==<br />
=== November 30, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Working on single instance code for MARIN ([https://github.com/VisTrails/VisTrails/pull/1205 #1205])<br />
** Remote shutdown ready ([https://github.com/VisTrails/VisTrails/pull/1204 #1204])<br />
** Still improving it<br />
** Want to return something more structured to the client (success/fail code, stdout, stderr)<br />
** Should be usable without Qt<br />
* [DK] Shutting down VisTrails has issues: cleans up interpreter (to remove files) but also resets persistent pipeline for no reason<br />
<br />
=== November 16, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [RR] Issue [https://github.com/VisTrails/VisTrails/issues/1200 #1200] from @samtux: need to set PyQt to API version 2 before ever importing it; using require_pyqt_api2() does that (also can install bundle)<br />
<br />
'''Items to Discuss'''<br />
<br />
* Email from Pieter: executing pipelines with parameters from command-line using single-instance<br />
** Seems to work, main instance gets full command-line and alters configuration temporarily<br />
** Not sure what his error is (VisTrails returns little information in a lot of cases); though execution error should return something more meaningful<br />
** Actions:<br />
*** Can add switch to kill main instance<br />
*** Can improve single-instance code to return more meaningful errors + machine-readable output from "client" VisTrails<br />
*** Add client/server code to <tt>core</tt> that doesn't use Qt? Also need a run.py-line entrypoint that doesn't use gui (calls console mode like <tt>gui.application.VistrailsApplicationSingleton#noninteractiveMode()</tt>)?<br />
<br />
=== November 2, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* New version 2.2-2 of Debian package, still [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=796994 wrong copyright], and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=748692 broken bundle installation]<br />
** Finally replied, says copyright issue will be fixed soon, working on including VisTrails 2.2.4 for Debian Stretch<br />
* Working on Docker package, low hanging fruit because we have CLTools, and useful for some workflow (also step towards CWL support)<br />
<br />
'''Items to Discuss'''<br />
<br />
=== October 5, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
<br />
* working toward a 3.0 release; [https://github.com/VisTrails/VisTrails/projects/1 project board]<br />
** unique ids<br />
** bundles<br />
** GUI fixes<br />
** wrapping updates<br />
** python3<br />
** interpreter?<br />
<br />
* command-line input parameters to provenance<br />
** [DK] we don't create a persisted workflow from those input params, i think<br />
** [DK] these are already defined as aliases so we treat them differently form a normal input param<br />
** [DK] still have the potential problem of allowing a user to view the workflow as it was run with a given set of aliases<br />
** Do the vistrails variables values get stored in provenance? (what if values change?)<br />
<br />
=== September 14, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] [http://aosabook.org/en/vistrails.html VisTrails] in [http://aosabook.org/ The Architecture of Open Source Applications] Vol I (2012)<br />
** Did we know/contribute to this?<br />
*** [DK] Yes. We wrote the text for the VisTrails chapter<br />
** Drafted response<br />
*** [DK] Looks good, will forward.<br />
<br />
=== August 31, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Went over vgc.poly.edu/vistrails.org config:<br />
** Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)<br />
** Fixed usersguide scripts (won't disappear again)<br />
** Moved some websites over to HTTPS<br />
** [https://docs.google.com/document/d/18YZewjR2YY9cQ9pzdA-nU9jbu2L7jYcPlDi3iaPadwA/edit Websites]<br />
* [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs ([http://vistrails.readthedocs.io/ currently there]) and users' guide under the same RTD "project"<br />
* [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that<br />
<br />
'''Items to Discuss'''<br />
* [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/<br />
** Get rid of more services? (svn, trac, ...)<br />
* [RR] Notebook execution works. How do we support more languages (than Python)?<br />
** Need: detection, metadata reader, code to input values<br />
* [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?<br />
<br />
=== August 17, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Fixed [https://github.com/VisTrails/VisTrails/commit/2cd42d156243067a7663bb0bf0156e5d87188c2f parameter order upgrade bug] that breaks VTK examples.<br />
* [TE] Matplotlib and sklearn are exportable using module-to-script api<br />
* [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.<br />
<br />
'''Items to Discuss'''<br />
* Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!<br />
* Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)<br />
* [TE] Wrap-up notes<br />
<br />
=== August 10, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Python Wrapper and scripting<br />
** All changes merged into python-wrapper branch<br />
*** Tests pass<br />
*** Still need to update sub-branches<br />
** Minor fixes<br />
*** vtkPlot3DReader upgrade bug<br />
*** Added tuple exporting<br />
** Most of vtk examples can be exported [https://gist.github.com/rexissimus/5ac7374cde8525d84f9443f019d234e2 example]<br />
*** Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary<br />
**** Could be a bug in the old vistrail, but not sure<br />
* [DK] Subworkflows and uuids<br />
<br />
=== August 3, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Wrapper patches now stored as [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/vtk_patch.py python code]<br />
** Connection variables now lowercase<br />
** Can we use code as output variables?<br />
*** E.g. "vtk_dataset_mapper.GetOutput(0)"<br />
*** Could mutate output when output method is only supposed to be called once<br />
*** Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)<br />
<br />
=== July 27, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] DB bundle serialization implemented<br />
<br />
'''Items to Discuss'''<br />
* [TE] Finished using patch in script [https://github.com/VisTrails/VisTrails/tree/python-wrapper-script-patches python-wrapper-script-patches]<br />
** locale?<br />
** [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d new patching example] vs [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 old patching example]<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/parse.py#L135-L273 Example of new patches] <br />
** Legacy '.vtkInstance' is now added as a patch "self.vtkInstance=self" (see new patching example)<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/api.py New api for executing standalone modules]<br />
*** Accessing module was disabled for core/api, but a different api for this is probably better anyway.<br />
*** api.Package now gives [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d#file-new_patching-py-L26 directly callable module compute functions]<br />
*** Only works on modules not using interpreter functionality ('.interpreter', etc.)<br />
**** for zip file modules and others that need temp files and use the interpreter for this, need to modify the module, but could work around with python libraries for managing temp files<br />
** Fixed more renaming issues - [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/scripts.py#L63-L64 function argument names should not be replaced]<br />
** Still need to do something nicer for the patch formatting<br />
*** patch keys currently contain non-python-variable characters but this can be fixed<br />
*** Need to replace string template with python variables that is replacable using RedBaron<br />
**** Patches could then look like:<br />
<br />
def vtkInstance():<br />
self.vtkInstance = self<br />
<br />
=== July 20, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] New patch method for python-wrapper [https://gist.github.com/rexissimus/009b303e4cadcd4b8f2fec6cf848cb2b example]<br />
** A patch is a string template with self/input/output variables for patching a class method <br />
** Patches are stored in xml specification<br />
** Patches are assigned to methods in ClassSpec's<br />
** Used both for execution in vistrails and when exporting as script<br />
*** Executed with "exec" when run in vistrails<br />
*** Turns patches into natural calls in script<br />
** Wrapping VTK library no longer needed<br />
*** Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)<br />
** Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)<br />
** Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)<br />
** TODO's<br />
*** Add to export_as_script<br />
*** Patch Modules that represent Python functions as well<br />
<br />
* [vistrails-users] Read data from SQL example?<br />
<br />
=== July 13, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] bundles on sql-alchemy branch<br />
** Bundle, BundleMapping, and BundleSerializer<br />
** New BaseSerializers to deal with versions of directory, zip, db serializers<br />
** Can register the BundleSerializers with the base serializers<br />
** Translation between bundle versions<br />
** Bundle.add_object "just works" if an appropriate mapping was registered<br />
<br />
'''Items to Discuss'''<br />
* [TE] Simplified running module using API for scripting [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 example]<br />
** API Module can be executed using '''compute''' method<br />
*** Takes list of outputs to compute + inputs as keyword arguments<br />
* Status of general cell synchronization (vistrails-users)<br />
<br />
=== July 6, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] Working on bundles<br />
<br />
'''Items to Discuss'''<br />
* [TE] Using vistrails API in scripting [https://gist.github.com/rexissimus/034d9c4dedc3299a079e84981dedd8c7 example 1 (API with VTK)] [https://gist.github.com/rexissimus/a4a9ca6c08d4058c64967d2b35415fd5 example 2 (DownloadFile only)]<br />
** Call module in script using vistrails API<br />
** Can be used when no export method is defined.<br />
* [TE] [https://github.com/VisTrails/VisTrails/commit/c76c109349769fee1280f85c22678380a2f9cd39 Server news bug] Critical?<br />
**[RR] No, unreleased<br />
* [DK] New bundle organization<br />
** BundleMapping contains individual mappings from raw objects to BundleObj<br />
** Must be separate in order to have in-memory, separate from serializer settings<br />
<br />
=== June 29, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as python script<br />
** Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) [https://gist.github.com/rexissimus/16caf5200802e150296f62a0c7844c5a example]<br />
** Preludes (imports) still need improvements<br />
** Works with vtkInteractionHandler (clipping plane)<br />
** Resulting clipping plane is incorrect? (supposed to hide one side of the plane)<br />
** vtkRendererOutput Uses VTKRenderWindow as output<br />
*** Should support different output modes?<br />
*** Does not work with multiple outputs (Next cell is displayed when current is closed)<br />
*** Uses wrapped vtkInstance (as vvtk)<br />
**** Create new patch system with code that can be used both in vistrails and in a python script?<br />
<br />
=== June 22, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fix for upgraded analogies (PR [https://github.com/VisTrails/VisTrails/pull/1177 #1177])<br />
** Uses module location to find module remap (The best we can do right now?)<br />
** Also fixes remapped port names for connections to existing modules <br />
** May still fail on complex upgrades like adding a function on an upgraded module that has a different port name<br />
<br />
* [TE] Working on exporting wrapped libraries (VTK)<br />
<br />
=== June 15, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Added depth logic to python script exporter ([https://github.com/VisTrails/VisTrails/commit/50f220e4f6abe5a7bb8689f73f456f43df2dbc12 full commit])<br />
** Fully implements connection merging and list looping<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Making analogy work on upgraded workflows ([https://github.com/VisTrails/VisTrails/issues/1175 #1175])<br />
** Analogy only works We need to use the non-upgraded action chain<br />
** But pipelines need to be valid (matcher uses module portspecs)<br />
** Tried to get upgrade remap from upgrade action, but this seems impossible<br />
** Use module location instead?<br />
*** Location should be identical for one-to-one module upgrades<br />
<br />
=== June 1, 2016 ===<br />
<br />
'''Updates'''<br />
* VisTrails 2.2.4 announced (finally)<br />
** 5 survey replies<br />
** 47 downloads last week<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as script (python-wrapper)<br />
** to_python_script<br />
*** needs the pipeline (or at least the connections)<br />
**** Should not generate code for any possible VTK method (exporter detects which ports is used in the source)<br />
**** VTK needs to know connection order<br />
**** Adding functions directly to the script results in nicer code<br />
** Add support for multiple input connections<br />
** Add support for list looping<br />
<br />
=== May 25, 2016 ===<br />
<br />
'''Updates'''<br />
* Mailing-list fixed<br />
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])<br />
<br />
'''Items to Discuss'''<br />
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)<br />
* [TE] Union ports<br />
** Show union ports in pipeline (DEMO)<br />
<br />
=== May 18, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] mailing list still down - Julio working on it<br />
* [TE] Improved union port<br />
** New schema 1.0.5 adds portitem.union string attribute<br />
** Works with pipeline view, parameter exploration view, and mashup view<br />
** Should work well for matplotlib<br />
** Use with VTK ports?<br />
*** It uses [item_1,item_2] style ports usually with different tuple length.<br />
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]<br />
<br />
=== May 11, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?<br />
*** Shows single port for all ports with same sort_key<br />
*** Select specific type when adding/showing<br />
*** Only one type should be visible on the module at any time<br />
*** Making tuple parameters union types is probably too complicated<br />
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly) <br />
<br />
* [TE] Feature suggestion: Let connections connect to functions/parameters<br />
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters<br />
** Use separate views for ports and functions (like before)<br />
*** Allow rearranging of functions<br />
** Pros:<br />
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.<br />
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.<br />
**** Makes List module simpler<br />
*** Connect to parameter in tuple<br />
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds. <br />
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.<br />
** Cons:<br />
*** Interface more complicated and crowded<br />
*** More difficult for users to grasp?<br />
*** No immediate use case<br />
*** Makes no union type for tuple parameters a bigger problem.<br />
<br />
* [TE] Publish user survey?<br />
** No response from vistrails release yet<br />
** Not announced on vistrails-users yet.<br />
<br />
=== May 4, 2016 ===<br />
'''Updates'''<br />
* VisTrails 2.2.4 released<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
<br />
=== April 27, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Wrapping Bokeh<br />
** New method type "operation"<br />
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output<br />
**** Glyph sets function name and arguments as output<br />
***** Not serializable, but should always be computed together with the subject of the operation.<br />
**** Figure applies Glyph operation on itself<br />
*** Useful for matplotlib?<br />
**** matplotlib currently find actors by searching upstream?<br />
**** But this is more difficult since matplotlib is not function-based.<br />
** New "union" port type<br />
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value<br />
**** Uses only first value in tuple<br />
**** Bokeh have many properties that support (list, string, Float)<br />
*** Does not work with depth=1 types (e.g. Float of depth 1)<br />
**** Turn Depth 1 types with known type into lists?<br />
<br />
* [TE] New bugfix release<br />
** Finalize survey<br />
<br />
=== April 20, 2016 ===<br />
'''Updates'''<br />
* Merged upgrade fixes<br />
** Now has infinite loop test<br />
** loop test now correctly loops 5 times<br />
*** remap_module used wrong (current) package version when creating ModuleDescriptor<br />
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])<br />
<br />
'''Items to Discuss'''<br />
* New bugfix release<br />
** Merge usagestats<br />
** Need to finalize survey<br />
<br />
=== April 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?<br />
<br />
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])<br />
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]<br />
** Cherry-pick test to preserve source?<br />
** Need maxPipelineFixAttempts option?<br />
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.<br />
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?<br />
*** Would need to be moved into handle_invalid_pipeline<br />
<br />
=== April 6, 2016 ===<br />
'''Updates'''<br />
* [TE] Query View improvements<br />
** Done except for displaying "Version Info" and #1164<br />
* [RR] Working on finishing new interpreter (finally!)<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Try latest upgrade first, then previous.<br />
** If all fail, select latest upgrade?<br />
<br />
=== March 30, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Query View improvements<br />
** Unified do_version_switch and get_upgrades as validate_version()<br />
** Problems with query view using its own controller<br />
*** Fixed module info not being displayed for query results<br />
*** Still no version info for query results<br />
*** Still an issue with version query view not being updated<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== Older meetings ==<br />
* [[Development/2015]]<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=12800Development2016-11-30T15:48:54Z<p>Remi: /* November 30, 2016 */ Single-instance code</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2016 ==<br />
=== November 30, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Working on single instance code for MARIN ([https://github.com/VisTrails/VisTrails/pull/1205 #1205])<br />
** Remote shutdown ready ([https://github.com/VisTrails/VisTrails/pull/1204 #1204])<br />
** Still improving it<br />
** Want to return something more structured to the client (success/fail code, stdout, stderr)<br />
** Should be usable without Qt<br />
<br />
=== November 16, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [RR] Issue [https://github.com/VisTrails/VisTrails/issues/1200 #1200] from @samtux: need to set PyQt to API version 2 before ever importing it; using require_pyqt_api2() does that (also can install bundle)<br />
<br />
'''Items to Discuss'''<br />
<br />
* Email from Pieter: executing pipelines with parameters from command-line using single-instance<br />
** Seems to work, main instance gets full command-line and alters configuration temporarily<br />
** Not sure what his error is (VisTrails returns little information in a lot of cases); though execution error should return something more meaningful<br />
** Actions:<br />
*** Can add switch to kill main instance<br />
*** Can improve single-instance code to return more meaningful errors + machine-readable output from "client" VisTrails<br />
*** Add client/server code to <tt>core</tt> that doesn't use Qt? Also need a run.py-line entrypoint that doesn't use gui (calls console mode like <tt>gui.application.VistrailsApplicationSingleton#noninteractiveMode()</tt>)?<br />
<br />
=== November 2, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* New version 2.2-2 of Debian package, still [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=796994 wrong copyright], and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=748692 broken bundle installation]<br />
** Finally replied, says copyright issue will be fixed soon, working on including VisTrails 2.2.4 for Debian Stretch<br />
* Working on Docker package, low hanging fruit because we have CLTools, and useful for some workflow (also step towards CWL support)<br />
<br />
'''Items to Discuss'''<br />
<br />
=== October 5, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
<br />
* working toward a 3.0 release; [https://github.com/VisTrails/VisTrails/projects/1 project board]<br />
** unique ids<br />
** bundles<br />
** GUI fixes<br />
** wrapping updates<br />
** python3<br />
** interpreter?<br />
<br />
* command-line input parameters to provenance<br />
** [DK] we don't create a persisted workflow from those input params, i think<br />
** [DK] these are already defined as aliases so we treat them differently form a normal input param<br />
** [DK] still have the potential problem of allowing a user to view the workflow as it was run with a given set of aliases<br />
** Do the vistrails variables values get stored in provenance? (what if values change?)<br />
<br />
=== September 14, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] [http://aosabook.org/en/vistrails.html VisTrails] in [http://aosabook.org/ The Architecture of Open Source Applications] Vol I (2012)<br />
** Did we know/contribute to this?<br />
*** [DK] Yes. We wrote the text for the VisTrails chapter<br />
** Drafted response<br />
*** [DK] Looks good, will forward.<br />
<br />
=== August 31, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Went over vgc.poly.edu/vistrails.org config:<br />
** Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)<br />
** Fixed usersguide scripts (won't disappear again)<br />
** Moved some websites over to HTTPS<br />
** [https://docs.google.com/document/d/18YZewjR2YY9cQ9pzdA-nU9jbu2L7jYcPlDi3iaPadwA/edit Websites]<br />
* [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs ([http://vistrails.readthedocs.io/ currently there]) and users' guide under the same RTD "project"<br />
* [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that<br />
<br />
'''Items to Discuss'''<br />
* [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/<br />
** Get rid of more services? (svn, trac, ...)<br />
* [RR] Notebook execution works. How do we support more languages (than Python)?<br />
** Need: detection, metadata reader, code to input values<br />
* [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?<br />
<br />
=== August 17, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Fixed [https://github.com/VisTrails/VisTrails/commit/2cd42d156243067a7663bb0bf0156e5d87188c2f parameter order upgrade bug] that breaks VTK examples.<br />
* [TE] Matplotlib and sklearn are exportable using module-to-script api<br />
* [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.<br />
<br />
'''Items to Discuss'''<br />
* Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!<br />
* Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)<br />
* [TE] Wrap-up notes<br />
<br />
=== August 10, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Python Wrapper and scripting<br />
** All changes merged into python-wrapper branch<br />
*** Tests pass<br />
*** Still need to update sub-branches<br />
** Minor fixes<br />
*** vtkPlot3DReader upgrade bug<br />
*** Added tuple exporting<br />
** Most of vtk examples can be exported [https://gist.github.com/rexissimus/5ac7374cde8525d84f9443f019d234e2 example]<br />
*** Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary<br />
**** Could be a bug in the old vistrail, but not sure<br />
* [DK] Subworkflows and uuids<br />
<br />
=== August 3, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Wrapper patches now stored as [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/vtk_patch.py python code]<br />
** Connection variables now lowercase<br />
** Can we use code as output variables?<br />
*** E.g. "vtk_dataset_mapper.GetOutput(0)"<br />
*** Could mutate output when output method is only supposed to be called once<br />
*** Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)<br />
<br />
=== July 27, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] DB bundle serialization implemented<br />
<br />
'''Items to Discuss'''<br />
* [TE] Finished using patch in script [https://github.com/VisTrails/VisTrails/tree/python-wrapper-script-patches python-wrapper-script-patches]<br />
** locale?<br />
** [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d new patching example] vs [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 old patching example]<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/parse.py#L135-L273 Example of new patches] <br />
** Legacy '.vtkInstance' is now added as a patch "self.vtkInstance=self" (see new patching example)<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/api.py New api for executing standalone modules]<br />
*** Accessing module was disabled for core/api, but a different api for this is probably better anyway.<br />
*** api.Package now gives [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d#file-new_patching-py-L26 directly callable module compute functions]<br />
*** Only works on modules not using interpreter functionality ('.interpreter', etc.)<br />
**** for zip file modules and others that need temp files and use the interpreter for this, need to modify the module, but could work around with python libraries for managing temp files<br />
** Fixed more renaming issues - [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/scripts.py#L63-L64 function argument names should not be replaced]<br />
** Still need to do something nicer for the patch formatting<br />
*** patch keys currently contain non-python-variable characters but this can be fixed<br />
*** Need to replace string template with python variables that is replacable using RedBaron<br />
**** Patches could then look like:<br />
<br />
def vtkInstance():<br />
self.vtkInstance = self<br />
<br />
=== July 20, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] New patch method for python-wrapper [https://gist.github.com/rexissimus/009b303e4cadcd4b8f2fec6cf848cb2b example]<br />
** A patch is a string template with self/input/output variables for patching a class method <br />
** Patches are stored in xml specification<br />
** Patches are assigned to methods in ClassSpec's<br />
** Used both for execution in vistrails and when exporting as script<br />
*** Executed with "exec" when run in vistrails<br />
*** Turns patches into natural calls in script<br />
** Wrapping VTK library no longer needed<br />
*** Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)<br />
** Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)<br />
** Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)<br />
** TODO's<br />
*** Add to export_as_script<br />
*** Patch Modules that represent Python functions as well<br />
<br />
* [vistrails-users] Read data from SQL example?<br />
<br />
=== July 13, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] bundles on sql-alchemy branch<br />
** Bundle, BundleMapping, and BundleSerializer<br />
** New BaseSerializers to deal with versions of directory, zip, db serializers<br />
** Can register the BundleSerializers with the base serializers<br />
** Translation between bundle versions<br />
** Bundle.add_object "just works" if an appropriate mapping was registered<br />
<br />
'''Items to Discuss'''<br />
* [TE] Simplified running module using API for scripting [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 example]<br />
** API Module can be executed using '''compute''' method<br />
*** Takes list of outputs to compute + inputs as keyword arguments<br />
* Status of general cell synchronization (vistrails-users)<br />
<br />
=== July 6, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] Working on bundles<br />
<br />
'''Items to Discuss'''<br />
* [TE] Using vistrails API in scripting [https://gist.github.com/rexissimus/034d9c4dedc3299a079e84981dedd8c7 example 1 (API with VTK)] [https://gist.github.com/rexissimus/a4a9ca6c08d4058c64967d2b35415fd5 example 2 (DownloadFile only)]<br />
** Call module in script using vistrails API<br />
** Can be used when no export method is defined.<br />
* [TE] [https://github.com/VisTrails/VisTrails/commit/c76c109349769fee1280f85c22678380a2f9cd39 Server news bug] Critical?<br />
**[RR] No, unreleased<br />
* [DK] New bundle organization<br />
** BundleMapping contains individual mappings from raw objects to BundleObj<br />
** Must be separate in order to have in-memory, separate from serializer settings<br />
<br />
=== June 29, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as python script<br />
** Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) [https://gist.github.com/rexissimus/16caf5200802e150296f62a0c7844c5a example]<br />
** Preludes (imports) still need improvements<br />
** Works with vtkInteractionHandler (clipping plane)<br />
** Resulting clipping plane is incorrect? (supposed to hide one side of the plane)<br />
** vtkRendererOutput Uses VTKRenderWindow as output<br />
*** Should support different output modes?<br />
*** Does not work with multiple outputs (Next cell is displayed when current is closed)<br />
*** Uses wrapped vtkInstance (as vvtk)<br />
**** Create new patch system with code that can be used both in vistrails and in a python script?<br />
<br />
=== June 22, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fix for upgraded analogies (PR [https://github.com/VisTrails/VisTrails/pull/1177 #1177])<br />
** Uses module location to find module remap (The best we can do right now?)<br />
** Also fixes remapped port names for connections to existing modules <br />
** May still fail on complex upgrades like adding a function on an upgraded module that has a different port name<br />
<br />
* [TE] Working on exporting wrapped libraries (VTK)<br />
<br />
=== June 15, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Added depth logic to python script exporter ([https://github.com/VisTrails/VisTrails/commit/50f220e4f6abe5a7bb8689f73f456f43df2dbc12 full commit])<br />
** Fully implements connection merging and list looping<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Making analogy work on upgraded workflows ([https://github.com/VisTrails/VisTrails/issues/1175 #1175])<br />
** Analogy only works We need to use the non-upgraded action chain<br />
** But pipelines need to be valid (matcher uses module portspecs)<br />
** Tried to get upgrade remap from upgrade action, but this seems impossible<br />
** Use module location instead?<br />
*** Location should be identical for one-to-one module upgrades<br />
<br />
=== June 1, 2016 ===<br />
<br />
'''Updates'''<br />
* VisTrails 2.2.4 announced (finally)<br />
** 5 survey replies<br />
** 47 downloads last week<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as script (python-wrapper)<br />
** to_python_script<br />
*** needs the pipeline (or at least the connections)<br />
**** Should not generate code for any possible VTK method (exporter detects which ports is used in the source)<br />
**** VTK needs to know connection order<br />
**** Adding functions directly to the script results in nicer code<br />
** Add support for multiple input connections<br />
** Add support for list looping<br />
<br />
=== May 25, 2016 ===<br />
<br />
'''Updates'''<br />
* Mailing-list fixed<br />
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])<br />
<br />
'''Items to Discuss'''<br />
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)<br />
* [TE] Union ports<br />
** Show union ports in pipeline (DEMO)<br />
<br />
=== May 18, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] mailing list still down - Julio working on it<br />
* [TE] Improved union port<br />
** New schema 1.0.5 adds portitem.union string attribute<br />
** Works with pipeline view, parameter exploration view, and mashup view<br />
** Should work well for matplotlib<br />
** Use with VTK ports?<br />
*** It uses [item_1,item_2] style ports usually with different tuple length.<br />
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]<br />
<br />
=== May 11, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?<br />
*** Shows single port for all ports with same sort_key<br />
*** Select specific type when adding/showing<br />
*** Only one type should be visible on the module at any time<br />
*** Making tuple parameters union types is probably too complicated<br />
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly) <br />
<br />
* [TE] Feature suggestion: Let connections connect to functions/parameters<br />
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters<br />
** Use separate views for ports and functions (like before)<br />
*** Allow rearranging of functions<br />
** Pros:<br />
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.<br />
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.<br />
**** Makes List module simpler<br />
*** Connect to parameter in tuple<br />
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds. <br />
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.<br />
** Cons:<br />
*** Interface more complicated and crowded<br />
*** More difficult for users to grasp?<br />
*** No immediate use case<br />
*** Makes no union type for tuple parameters a bigger problem.<br />
<br />
* [TE] Publish user survey?<br />
** No response from vistrails release yet<br />
** Not announced on vistrails-users yet.<br />
<br />
=== May 4, 2016 ===<br />
'''Updates'''<br />
* VisTrails 2.2.4 released<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
<br />
=== April 27, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Wrapping Bokeh<br />
** New method type "operation"<br />
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output<br />
**** Glyph sets function name and arguments as output<br />
***** Not serializable, but should always be computed together with the subject of the operation.<br />
**** Figure applies Glyph operation on itself<br />
*** Useful for matplotlib?<br />
**** matplotlib currently find actors by searching upstream?<br />
**** But this is more difficult since matplotlib is not function-based.<br />
** New "union" port type<br />
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value<br />
**** Uses only first value in tuple<br />
**** Bokeh have many properties that support (list, string, Float)<br />
*** Does not work with depth=1 types (e.g. Float of depth 1)<br />
**** Turn Depth 1 types with known type into lists?<br />
<br />
* [TE] New bugfix release<br />
** Finalize survey<br />
<br />
=== April 20, 2016 ===<br />
'''Updates'''<br />
* Merged upgrade fixes<br />
** Now has infinite loop test<br />
** loop test now correctly loops 5 times<br />
*** remap_module used wrong (current) package version when creating ModuleDescriptor<br />
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])<br />
<br />
'''Items to Discuss'''<br />
* New bugfix release<br />
** Merge usagestats<br />
** Need to finalize survey<br />
<br />
=== April 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?<br />
<br />
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])<br />
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]<br />
** Cherry-pick test to preserve source?<br />
** Need maxPipelineFixAttempts option?<br />
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.<br />
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?<br />
*** Would need to be moved into handle_invalid_pipeline<br />
<br />
=== April 6, 2016 ===<br />
'''Updates'''<br />
* [TE] Query View improvements<br />
** Done except for displaying "Version Info" and #1164<br />
* [RR] Working on finishing new interpreter (finally!)<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Try latest upgrade first, then previous.<br />
** If all fail, select latest upgrade?<br />
<br />
=== March 30, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Query View improvements<br />
** Unified do_version_switch and get_upgrades as validate_version()<br />
** Problems with query view using its own controller<br />
*** Fixed module info not being displayed for query results<br />
*** Still no version info for query results<br />
*** Still an issue with version query view not being updated<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== Older meetings ==<br />
* [[Development/2015]]<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=12799Development2016-11-30T15:41:20Z<p>Remi: /* November 30, 2016 */</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2016 ==<br />
=== November 30, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
=== November 16, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [RR] Issue [https://github.com/VisTrails/VisTrails/issues/1200 #1200] from @samtux: need to set PyQt to API version 2 before ever importing it; using require_pyqt_api2() does that (also can install bundle)<br />
<br />
'''Items to Discuss'''<br />
<br />
* Email from Pieter: executing pipelines with parameters from command-line using single-instance<br />
** Seems to work, main instance gets full command-line and alters configuration temporarily<br />
** Not sure what his error is (VisTrails returns little information in a lot of cases); though execution error should return something more meaningful<br />
** Actions:<br />
*** Can add switch to kill main instance<br />
*** Can improve single-instance code to return more meaningful errors + machine-readable output from "client" VisTrails<br />
*** Add client/server code to <tt>core</tt> that doesn't use Qt? Also need a run.py-line entrypoint that doesn't use gui (calls console mode like <tt>gui.application.VistrailsApplicationSingleton#noninteractiveMode()</tt>)?<br />
<br />
=== November 2, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* New version 2.2-2 of Debian package, still [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=796994 wrong copyright], and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=748692 broken bundle installation]<br />
** Finally replied, says copyright issue will be fixed soon, working on including VisTrails 2.2.4 for Debian Stretch<br />
* Working on Docker package, low hanging fruit because we have CLTools, and useful for some workflow (also step towards CWL support)<br />
<br />
'''Items to Discuss'''<br />
<br />
=== October 5, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
<br />
* working toward a 3.0 release; [https://github.com/VisTrails/VisTrails/projects/1 project board]<br />
** unique ids<br />
** bundles<br />
** GUI fixes<br />
** wrapping updates<br />
** python3<br />
** interpreter?<br />
<br />
* command-line input parameters to provenance<br />
** [DK] we don't create a persisted workflow from those input params, i think<br />
** [DK] these are already defined as aliases so we treat them differently form a normal input param<br />
** [DK] still have the potential problem of allowing a user to view the workflow as it was run with a given set of aliases<br />
** Do the vistrails variables values get stored in provenance? (what if values change?)<br />
<br />
=== September 14, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] [http://aosabook.org/en/vistrails.html VisTrails] in [http://aosabook.org/ The Architecture of Open Source Applications] Vol I (2012)<br />
** Did we know/contribute to this?<br />
*** [DK] Yes. We wrote the text for the VisTrails chapter<br />
** Drafted response<br />
*** [DK] Looks good, will forward.<br />
<br />
=== August 31, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Went over vgc.poly.edu/vistrails.org config:<br />
** Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)<br />
** Fixed usersguide scripts (won't disappear again)<br />
** Moved some websites over to HTTPS<br />
** [https://docs.google.com/document/d/18YZewjR2YY9cQ9pzdA-nU9jbu2L7jYcPlDi3iaPadwA/edit Websites]<br />
* [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs ([http://vistrails.readthedocs.io/ currently there]) and users' guide under the same RTD "project"<br />
* [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that<br />
<br />
'''Items to Discuss'''<br />
* [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/<br />
** Get rid of more services? (svn, trac, ...)<br />
* [RR] Notebook execution works. How do we support more languages (than Python)?<br />
** Need: detection, metadata reader, code to input values<br />
* [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?<br />
<br />
=== August 17, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Fixed [https://github.com/VisTrails/VisTrails/commit/2cd42d156243067a7663bb0bf0156e5d87188c2f parameter order upgrade bug] that breaks VTK examples.<br />
* [TE] Matplotlib and sklearn are exportable using module-to-script api<br />
* [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.<br />
<br />
'''Items to Discuss'''<br />
* Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!<br />
* Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)<br />
* [TE] Wrap-up notes<br />
<br />
=== August 10, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Python Wrapper and scripting<br />
** All changes merged into python-wrapper branch<br />
*** Tests pass<br />
*** Still need to update sub-branches<br />
** Minor fixes<br />
*** vtkPlot3DReader upgrade bug<br />
*** Added tuple exporting<br />
** Most of vtk examples can be exported [https://gist.github.com/rexissimus/5ac7374cde8525d84f9443f019d234e2 example]<br />
*** Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary<br />
**** Could be a bug in the old vistrail, but not sure<br />
* [DK] Subworkflows and uuids<br />
<br />
=== August 3, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Wrapper patches now stored as [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/vtk_patch.py python code]<br />
** Connection variables now lowercase<br />
** Can we use code as output variables?<br />
*** E.g. "vtk_dataset_mapper.GetOutput(0)"<br />
*** Could mutate output when output method is only supposed to be called once<br />
*** Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)<br />
<br />
=== July 27, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] DB bundle serialization implemented<br />
<br />
'''Items to Discuss'''<br />
* [TE] Finished using patch in script [https://github.com/VisTrails/VisTrails/tree/python-wrapper-script-patches python-wrapper-script-patches]<br />
** locale?<br />
** [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d new patching example] vs [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 old patching example]<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/parse.py#L135-L273 Example of new patches] <br />
** Legacy '.vtkInstance' is now added as a patch "self.vtkInstance=self" (see new patching example)<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/api.py New api for executing standalone modules]<br />
*** Accessing module was disabled for core/api, but a different api for this is probably better anyway.<br />
*** api.Package now gives [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d#file-new_patching-py-L26 directly callable module compute functions]<br />
*** Only works on modules not using interpreter functionality ('.interpreter', etc.)<br />
**** for zip file modules and others that need temp files and use the interpreter for this, need to modify the module, but could work around with python libraries for managing temp files<br />
** Fixed more renaming issues - [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/scripts.py#L63-L64 function argument names should not be replaced]<br />
** Still need to do something nicer for the patch formatting<br />
*** patch keys currently contain non-python-variable characters but this can be fixed<br />
*** Need to replace string template with python variables that is replacable using RedBaron<br />
**** Patches could then look like:<br />
<br />
def vtkInstance():<br />
self.vtkInstance = self<br />
<br />
=== July 20, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] New patch method for python-wrapper [https://gist.github.com/rexissimus/009b303e4cadcd4b8f2fec6cf848cb2b example]<br />
** A patch is a string template with self/input/output variables for patching a class method <br />
** Patches are stored in xml specification<br />
** Patches are assigned to methods in ClassSpec's<br />
** Used both for execution in vistrails and when exporting as script<br />
*** Executed with "exec" when run in vistrails<br />
*** Turns patches into natural calls in script<br />
** Wrapping VTK library no longer needed<br />
*** Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)<br />
** Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)<br />
** Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)<br />
** TODO's<br />
*** Add to export_as_script<br />
*** Patch Modules that represent Python functions as well<br />
<br />
* [vistrails-users] Read data from SQL example?<br />
<br />
=== July 13, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] bundles on sql-alchemy branch<br />
** Bundle, BundleMapping, and BundleSerializer<br />
** New BaseSerializers to deal with versions of directory, zip, db serializers<br />
** Can register the BundleSerializers with the base serializers<br />
** Translation between bundle versions<br />
** Bundle.add_object "just works" if an appropriate mapping was registered<br />
<br />
'''Items to Discuss'''<br />
* [TE] Simplified running module using API for scripting [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 example]<br />
** API Module can be executed using '''compute''' method<br />
*** Takes list of outputs to compute + inputs as keyword arguments<br />
* Status of general cell synchronization (vistrails-users)<br />
<br />
=== July 6, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] Working on bundles<br />
<br />
'''Items to Discuss'''<br />
* [TE] Using vistrails API in scripting [https://gist.github.com/rexissimus/034d9c4dedc3299a079e84981dedd8c7 example 1 (API with VTK)] [https://gist.github.com/rexissimus/a4a9ca6c08d4058c64967d2b35415fd5 example 2 (DownloadFile only)]<br />
** Call module in script using vistrails API<br />
** Can be used when no export method is defined.<br />
* [TE] [https://github.com/VisTrails/VisTrails/commit/c76c109349769fee1280f85c22678380a2f9cd39 Server news bug] Critical?<br />
**[RR] No, unreleased<br />
* [DK] New bundle organization<br />
** BundleMapping contains individual mappings from raw objects to BundleObj<br />
** Must be separate in order to have in-memory, separate from serializer settings<br />
<br />
=== June 29, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as python script<br />
** Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) [https://gist.github.com/rexissimus/16caf5200802e150296f62a0c7844c5a example]<br />
** Preludes (imports) still need improvements<br />
** Works with vtkInteractionHandler (clipping plane)<br />
** Resulting clipping plane is incorrect? (supposed to hide one side of the plane)<br />
** vtkRendererOutput Uses VTKRenderWindow as output<br />
*** Should support different output modes?<br />
*** Does not work with multiple outputs (Next cell is displayed when current is closed)<br />
*** Uses wrapped vtkInstance (as vvtk)<br />
**** Create new patch system with code that can be used both in vistrails and in a python script?<br />
<br />
=== June 22, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fix for upgraded analogies (PR [https://github.com/VisTrails/VisTrails/pull/1177 #1177])<br />
** Uses module location to find module remap (The best we can do right now?)<br />
** Also fixes remapped port names for connections to existing modules <br />
** May still fail on complex upgrades like adding a function on an upgraded module that has a different port name<br />
<br />
* [TE] Working on exporting wrapped libraries (VTK)<br />
<br />
=== June 15, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Added depth logic to python script exporter ([https://github.com/VisTrails/VisTrails/commit/50f220e4f6abe5a7bb8689f73f456f43df2dbc12 full commit])<br />
** Fully implements connection merging and list looping<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Making analogy work on upgraded workflows ([https://github.com/VisTrails/VisTrails/issues/1175 #1175])<br />
** Analogy only works We need to use the non-upgraded action chain<br />
** But pipelines need to be valid (matcher uses module portspecs)<br />
** Tried to get upgrade remap from upgrade action, but this seems impossible<br />
** Use module location instead?<br />
*** Location should be identical for one-to-one module upgrades<br />
<br />
=== June 1, 2016 ===<br />
<br />
'''Updates'''<br />
* VisTrails 2.2.4 announced (finally)<br />
** 5 survey replies<br />
** 47 downloads last week<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as script (python-wrapper)<br />
** to_python_script<br />
*** needs the pipeline (or at least the connections)<br />
**** Should not generate code for any possible VTK method (exporter detects which ports is used in the source)<br />
**** VTK needs to know connection order<br />
**** Adding functions directly to the script results in nicer code<br />
** Add support for multiple input connections<br />
** Add support for list looping<br />
<br />
=== May 25, 2016 ===<br />
<br />
'''Updates'''<br />
* Mailing-list fixed<br />
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])<br />
<br />
'''Items to Discuss'''<br />
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)<br />
* [TE] Union ports<br />
** Show union ports in pipeline (DEMO)<br />
<br />
=== May 18, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] mailing list still down - Julio working on it<br />
* [TE] Improved union port<br />
** New schema 1.0.5 adds portitem.union string attribute<br />
** Works with pipeline view, parameter exploration view, and mashup view<br />
** Should work well for matplotlib<br />
** Use with VTK ports?<br />
*** It uses [item_1,item_2] style ports usually with different tuple length.<br />
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]<br />
<br />
=== May 11, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?<br />
*** Shows single port for all ports with same sort_key<br />
*** Select specific type when adding/showing<br />
*** Only one type should be visible on the module at any time<br />
*** Making tuple parameters union types is probably too complicated<br />
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly) <br />
<br />
* [TE] Feature suggestion: Let connections connect to functions/parameters<br />
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters<br />
** Use separate views for ports and functions (like before)<br />
*** Allow rearranging of functions<br />
** Pros:<br />
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.<br />
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.<br />
**** Makes List module simpler<br />
*** Connect to parameter in tuple<br />
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds. <br />
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.<br />
** Cons:<br />
*** Interface more complicated and crowded<br />
*** More difficult for users to grasp?<br />
*** No immediate use case<br />
*** Makes no union type for tuple parameters a bigger problem.<br />
<br />
* [TE] Publish user survey?<br />
** No response from vistrails release yet<br />
** Not announced on vistrails-users yet.<br />
<br />
=== May 4, 2016 ===<br />
'''Updates'''<br />
* VisTrails 2.2.4 released<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
<br />
=== April 27, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Wrapping Bokeh<br />
** New method type "operation"<br />
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output<br />
**** Glyph sets function name and arguments as output<br />
***** Not serializable, but should always be computed together with the subject of the operation.<br />
**** Figure applies Glyph operation on itself<br />
*** Useful for matplotlib?<br />
**** matplotlib currently find actors by searching upstream?<br />
**** But this is more difficult since matplotlib is not function-based.<br />
** New "union" port type<br />
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value<br />
**** Uses only first value in tuple<br />
**** Bokeh have many properties that support (list, string, Float)<br />
*** Does not work with depth=1 types (e.g. Float of depth 1)<br />
**** Turn Depth 1 types with known type into lists?<br />
<br />
* [TE] New bugfix release<br />
** Finalize survey<br />
<br />
=== April 20, 2016 ===<br />
'''Updates'''<br />
* Merged upgrade fixes<br />
** Now has infinite loop test<br />
** loop test now correctly loops 5 times<br />
*** remap_module used wrong (current) package version when creating ModuleDescriptor<br />
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])<br />
<br />
'''Items to Discuss'''<br />
* New bugfix release<br />
** Merge usagestats<br />
** Need to finalize survey<br />
<br />
=== April 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?<br />
<br />
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])<br />
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]<br />
** Cherry-pick test to preserve source?<br />
** Need maxPipelineFixAttempts option?<br />
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.<br />
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?<br />
*** Would need to be moved into handle_invalid_pipeline<br />
<br />
=== April 6, 2016 ===<br />
'''Updates'''<br />
* [TE] Query View improvements<br />
** Done except for displaying "Version Info" and #1164<br />
* [RR] Working on finishing new interpreter (finally!)<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Try latest upgrade first, then previous.<br />
** If all fail, select latest upgrade?<br />
<br />
=== March 30, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Query View improvements<br />
** Unified do_version_switch and get_upgrades as validate_version()<br />
** Problems with query view using its own controller<br />
*** Fixed module info not being displayed for query results<br />
*** Still no version info for query results<br />
*** Still an issue with version query view not being updated<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== Older meetings ==<br />
* [[Development/2015]]<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=12704Development2016-11-16T15:35:56Z<p>Remi: /* November 16, 2016 */</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2016 ==<br />
=== November 16, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [RR] Issue [https://github.com/VisTrails/VisTrails/issues/1200 #1200] from @samtux: need to set PyQt to API version 2 before ever importing it; using require_pyqt_api2() does that (also can install bundle)<br />
<br />
'''Items to Discuss'''<br />
<br />
* Email from Pieter: executing pipelines with parameters from command-line using single-instance<br />
** Seems to work, main instance gets full command-line and alters configuration temporarily<br />
** Not sure what his error is (VisTrails returns little information in a lot of cases); though execution error should return something more meaningful<br />
** Actions:<br />
*** Can add switch to kill main instance<br />
*** Can improve single-instance code to return more meaningful errors + machine-readable output from "client" VisTrails<br />
*** Add client/server code to <tt>core</tt> that doesn't use Qt? Also need a run.py-line entrypoint that doesn't use gui (calls console mode like <tt>gui.application.VistrailsApplicationSingleton#noninteractiveMode()</tt>)?<br />
<br />
=== November 2, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* New version 2.2-2 of Debian package, still [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=796994 wrong copyright], and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=748692 broken bundle installation]<br />
** Finally replied, says copyright issue will be fixed soon, working on including VisTrails 2.2.4 for Debian Stretch<br />
* Working on Docker package, low hanging fruit because we have CLTools, and useful for some workflow (also step towards CWL support)<br />
<br />
'''Items to Discuss'''<br />
<br />
=== October 5, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
<br />
* working toward a 3.0 release; [https://github.com/VisTrails/VisTrails/projects/1 project board]<br />
** unique ids<br />
** bundles<br />
** GUI fixes<br />
** wrapping updates<br />
** python3<br />
** interpreter?<br />
<br />
* command-line input parameters to provenance<br />
** [DK] we don't create a persisted workflow from those input params, i think<br />
** [DK] these are already defined as aliases so we treat them differently form a normal input param<br />
** [DK] still have the potential problem of allowing a user to view the workflow as it was run with a given set of aliases<br />
** Do the vistrails variables values get stored in provenance? (what if values change?)<br />
<br />
=== September 14, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] [http://aosabook.org/en/vistrails.html VisTrails] in [http://aosabook.org/ The Architecture of Open Source Applications] Vol I (2012)<br />
** Did we know/contribute to this?<br />
*** [DK] Yes. We wrote the text for the VisTrails chapter<br />
** Drafted response<br />
*** [DK] Looks good, will forward.<br />
<br />
=== August 31, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Went over vgc.poly.edu/vistrails.org config:<br />
** Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)<br />
** Fixed usersguide scripts (won't disappear again)<br />
** Moved some websites over to HTTPS<br />
** [https://docs.google.com/document/d/18YZewjR2YY9cQ9pzdA-nU9jbu2L7jYcPlDi3iaPadwA/edit Websites]<br />
* [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs ([http://vistrails.readthedocs.io/ currently there]) and users' guide under the same RTD "project"<br />
* [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that<br />
<br />
'''Items to Discuss'''<br />
* [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/<br />
** Get rid of more services? (svn, trac, ...)<br />
* [RR] Notebook execution works. How do we support more languages (than Python)?<br />
** Need: detection, metadata reader, code to input values<br />
* [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?<br />
<br />
=== August 17, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Fixed [https://github.com/VisTrails/VisTrails/commit/2cd42d156243067a7663bb0bf0156e5d87188c2f parameter order upgrade bug] that breaks VTK examples.<br />
* [TE] Matplotlib and sklearn are exportable using module-to-script api<br />
* [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.<br />
<br />
'''Items to Discuss'''<br />
* Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!<br />
* Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)<br />
* [TE] Wrap-up notes<br />
<br />
=== August 10, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Python Wrapper and scripting<br />
** All changes merged into python-wrapper branch<br />
*** Tests pass<br />
*** Still need to update sub-branches<br />
** Minor fixes<br />
*** vtkPlot3DReader upgrade bug<br />
*** Added tuple exporting<br />
** Most of vtk examples can be exported [https://gist.github.com/rexissimus/5ac7374cde8525d84f9443f019d234e2 example]<br />
*** Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary<br />
**** Could be a bug in the old vistrail, but not sure<br />
* [DK] Subworkflows and uuids<br />
<br />
=== August 3, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Wrapper patches now stored as [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/vtk_patch.py python code]<br />
** Connection variables now lowercase<br />
** Can we use code as output variables?<br />
*** E.g. "vtk_dataset_mapper.GetOutput(0)"<br />
*** Could mutate output when output method is only supposed to be called once<br />
*** Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)<br />
<br />
=== July 27, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] DB bundle serialization implemented<br />
<br />
'''Items to Discuss'''<br />
* [TE] Finished using patch in script [https://github.com/VisTrails/VisTrails/tree/python-wrapper-script-patches python-wrapper-script-patches]<br />
** locale?<br />
** [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d new patching example] vs [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 old patching example]<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/parse.py#L135-L273 Example of new patches] <br />
** Legacy '.vtkInstance' is now added as a patch "self.vtkInstance=self" (see new patching example)<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/api.py New api for executing standalone modules]<br />
*** Accessing module was disabled for core/api, but a different api for this is probably better anyway.<br />
*** api.Package now gives [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d#file-new_patching-py-L26 directly callable module compute functions]<br />
*** Only works on modules not using interpreter functionality ('.interpreter', etc.)<br />
**** for zip file modules and others that need temp files and use the interpreter for this, need to modify the module, but could work around with python libraries for managing temp files<br />
** Fixed more renaming issues - [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/scripts.py#L63-L64 function argument names should not be replaced]<br />
** Still need to do something nicer for the patch formatting<br />
*** patch keys currently contain non-python-variable characters but this can be fixed<br />
*** Need to replace string template with python variables that is replacable using RedBaron<br />
**** Patches could then look like:<br />
<br />
def vtkInstance():<br />
self.vtkInstance = self<br />
<br />
=== July 20, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] New patch method for python-wrapper [https://gist.github.com/rexissimus/009b303e4cadcd4b8f2fec6cf848cb2b example]<br />
** A patch is a string template with self/input/output variables for patching a class method <br />
** Patches are stored in xml specification<br />
** Patches are assigned to methods in ClassSpec's<br />
** Used both for execution in vistrails and when exporting as script<br />
*** Executed with "exec" when run in vistrails<br />
*** Turns patches into natural calls in script<br />
** Wrapping VTK library no longer needed<br />
*** Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)<br />
** Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)<br />
** Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)<br />
** TODO's<br />
*** Add to export_as_script<br />
*** Patch Modules that represent Python functions as well<br />
<br />
* [vistrails-users] Read data from SQL example?<br />
<br />
=== July 13, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] bundles on sql-alchemy branch<br />
** Bundle, BundleMapping, and BundleSerializer<br />
** New BaseSerializers to deal with versions of directory, zip, db serializers<br />
** Can register the BundleSerializers with the base serializers<br />
** Translation between bundle versions<br />
** Bundle.add_object "just works" if an appropriate mapping was registered<br />
<br />
'''Items to Discuss'''<br />
* [TE] Simplified running module using API for scripting [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 example]<br />
** API Module can be executed using '''compute''' method<br />
*** Takes list of outputs to compute + inputs as keyword arguments<br />
* Status of general cell synchronization (vistrails-users)<br />
<br />
=== July 6, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] Working on bundles<br />
<br />
'''Items to Discuss'''<br />
* [TE] Using vistrails API in scripting [https://gist.github.com/rexissimus/034d9c4dedc3299a079e84981dedd8c7 example 1 (API with VTK)] [https://gist.github.com/rexissimus/a4a9ca6c08d4058c64967d2b35415fd5 example 2 (DownloadFile only)]<br />
** Call module in script using vistrails API<br />
** Can be used when no export method is defined.<br />
* [TE] [https://github.com/VisTrails/VisTrails/commit/c76c109349769fee1280f85c22678380a2f9cd39 Server news bug] Critical?<br />
**[RR] No, unreleased<br />
* [DK] New bundle organization<br />
** BundleMapping contains individual mappings from raw objects to BundleObj<br />
** Must be separate in order to have in-memory, separate from serializer settings<br />
<br />
=== June 29, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as python script<br />
** Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) [https://gist.github.com/rexissimus/16caf5200802e150296f62a0c7844c5a example]<br />
** Preludes (imports) still need improvements<br />
** Works with vtkInteractionHandler (clipping plane)<br />
** Resulting clipping plane is incorrect? (supposed to hide one side of the plane)<br />
** vtkRendererOutput Uses VTKRenderWindow as output<br />
*** Should support different output modes?<br />
*** Does not work with multiple outputs (Next cell is displayed when current is closed)<br />
*** Uses wrapped vtkInstance (as vvtk)<br />
**** Create new patch system with code that can be used both in vistrails and in a python script?<br />
<br />
=== June 22, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fix for upgraded analogies (PR [https://github.com/VisTrails/VisTrails/pull/1177 #1177])<br />
** Uses module location to find module remap (The best we can do right now?)<br />
** Also fixes remapped port names for connections to existing modules <br />
** May still fail on complex upgrades like adding a function on an upgraded module that has a different port name<br />
<br />
* [TE] Working on exporting wrapped libraries (VTK)<br />
<br />
=== June 15, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Added depth logic to python script exporter ([https://github.com/VisTrails/VisTrails/commit/50f220e4f6abe5a7bb8689f73f456f43df2dbc12 full commit])<br />
** Fully implements connection merging and list looping<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Making analogy work on upgraded workflows ([https://github.com/VisTrails/VisTrails/issues/1175 #1175])<br />
** Analogy only works We need to use the non-upgraded action chain<br />
** But pipelines need to be valid (matcher uses module portspecs)<br />
** Tried to get upgrade remap from upgrade action, but this seems impossible<br />
** Use module location instead?<br />
*** Location should be identical for one-to-one module upgrades<br />
<br />
=== June 1, 2016 ===<br />
<br />
'''Updates'''<br />
* VisTrails 2.2.4 announced (finally)<br />
** 5 survey replies<br />
** 47 downloads last week<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as script (python-wrapper)<br />
** to_python_script<br />
*** needs the pipeline (or at least the connections)<br />
**** Should not generate code for any possible VTK method (exporter detects which ports is used in the source)<br />
**** VTK needs to know connection order<br />
**** Adding functions directly to the script results in nicer code<br />
** Add support for multiple input connections<br />
** Add support for list looping<br />
<br />
=== May 25, 2016 ===<br />
<br />
'''Updates'''<br />
* Mailing-list fixed<br />
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])<br />
<br />
'''Items to Discuss'''<br />
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)<br />
* [TE] Union ports<br />
** Show union ports in pipeline (DEMO)<br />
<br />
=== May 18, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] mailing list still down - Julio working on it<br />
* [TE] Improved union port<br />
** New schema 1.0.5 adds portitem.union string attribute<br />
** Works with pipeline view, parameter exploration view, and mashup view<br />
** Should work well for matplotlib<br />
** Use with VTK ports?<br />
*** It uses [item_1,item_2] style ports usually with different tuple length.<br />
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]<br />
<br />
=== May 11, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?<br />
*** Shows single port for all ports with same sort_key<br />
*** Select specific type when adding/showing<br />
*** Only one type should be visible on the module at any time<br />
*** Making tuple parameters union types is probably too complicated<br />
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly) <br />
<br />
* [TE] Feature suggestion: Let connections connect to functions/parameters<br />
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters<br />
** Use separate views for ports and functions (like before)<br />
*** Allow rearranging of functions<br />
** Pros:<br />
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.<br />
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.<br />
**** Makes List module simpler<br />
*** Connect to parameter in tuple<br />
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds. <br />
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.<br />
** Cons:<br />
*** Interface more complicated and crowded<br />
*** More difficult for users to grasp?<br />
*** No immediate use case<br />
*** Makes no union type for tuple parameters a bigger problem.<br />
<br />
* [TE] Publish user survey?<br />
** No response from vistrails release yet<br />
** Not announced on vistrails-users yet.<br />
<br />
=== May 4, 2016 ===<br />
'''Updates'''<br />
* VisTrails 2.2.4 released<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
<br />
=== April 27, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Wrapping Bokeh<br />
** New method type "operation"<br />
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output<br />
**** Glyph sets function name and arguments as output<br />
***** Not serializable, but should always be computed together with the subject of the operation.<br />
**** Figure applies Glyph operation on itself<br />
*** Useful for matplotlib?<br />
**** matplotlib currently find actors by searching upstream?<br />
**** But this is more difficult since matplotlib is not function-based.<br />
** New "union" port type<br />
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value<br />
**** Uses only first value in tuple<br />
**** Bokeh have many properties that support (list, string, Float)<br />
*** Does not work with depth=1 types (e.g. Float of depth 1)<br />
**** Turn Depth 1 types with known type into lists?<br />
<br />
* [TE] New bugfix release<br />
** Finalize survey<br />
<br />
=== April 20, 2016 ===<br />
'''Updates'''<br />
* Merged upgrade fixes<br />
** Now has infinite loop test<br />
** loop test now correctly loops 5 times<br />
*** remap_module used wrong (current) package version when creating ModuleDescriptor<br />
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])<br />
<br />
'''Items to Discuss'''<br />
* New bugfix release<br />
** Merge usagestats<br />
** Need to finalize survey<br />
<br />
=== April 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?<br />
<br />
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])<br />
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]<br />
** Cherry-pick test to preserve source?<br />
** Need maxPipelineFixAttempts option?<br />
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.<br />
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?<br />
*** Would need to be moved into handle_invalid_pipeline<br />
<br />
=== April 6, 2016 ===<br />
'''Updates'''<br />
* [TE] Query View improvements<br />
** Done except for displaying "Version Info" and #1164<br />
* [RR] Working on finishing new interpreter (finally!)<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Try latest upgrade first, then previous.<br />
** If all fail, select latest upgrade?<br />
<br />
=== March 30, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Query View improvements<br />
** Unified do_version_switch and get_upgrades as validate_version()<br />
** Problems with query view using its own controller<br />
*** Fixed module info not being displayed for query results<br />
*** Still no version info for query results<br />
*** Still an issue with version query view not being updated<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== Older meetings ==<br />
* [[Development/2015]]<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=12623Development2016-11-02T19:40:49Z<p>Remi: /* November 2, 2016 */</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2016 ==<br />
=== November 2, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* New version 2.2-2 of Debian package, still [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=796994 wrong copyright], and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=748692 broken bundle installation]<br />
** Finally replied, says copyright issue will be fixed soon, working on including VisTrails 2.2.4 for Debian Stretch<br />
* Working on Docker package, low hanging fruit because we have CLTools, and useful for some workflow (also step towards CWL support)<br />
<br />
'''Items to Discuss'''<br />
<br />
=== October 5, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
<br />
* working toward a 3.0 release; [https://github.com/VisTrails/VisTrails/projects/1 project board]<br />
** unique ids<br />
** bundles<br />
** GUI fixes<br />
** wrapping updates<br />
** python3<br />
** interpreter?<br />
<br />
* command-line input parameters to provenance<br />
** [DK] we don't create a persisted workflow from those input params, i think<br />
** [DK] these are already defined as aliases so we treat them differently form a normal input param<br />
** [DK] still have the potential problem of allowing a user to view the workflow as it was run with a given set of aliases<br />
** Do the vistrails variables values get stored in provenance? (what if values change?)<br />
<br />
=== September 14, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] [http://aosabook.org/en/vistrails.html VisTrails] in [http://aosabook.org/ The Architecture of Open Source Applications] Vol I (2012)<br />
** Did we know/contribute to this?<br />
*** [DK] Yes. We wrote the text for the VisTrails chapter<br />
** Drafted response<br />
*** [DK] Looks good, will forward.<br />
<br />
=== August 31, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Went over vgc.poly.edu/vistrails.org config:<br />
** Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)<br />
** Fixed usersguide scripts (won't disappear again)<br />
** Moved some websites over to HTTPS<br />
** [https://docs.google.com/document/d/18YZewjR2YY9cQ9pzdA-nU9jbu2L7jYcPlDi3iaPadwA/edit Websites]<br />
* [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs ([http://vistrails.readthedocs.io/ currently there]) and users' guide under the same RTD "project"<br />
* [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that<br />
<br />
'''Items to Discuss'''<br />
* [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/<br />
** Get rid of more services? (svn, trac, ...)<br />
* [RR] Notebook execution works. How do we support more languages (than Python)?<br />
** Need: detection, metadata reader, code to input values<br />
* [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?<br />
<br />
=== August 17, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Fixed [https://github.com/VisTrails/VisTrails/commit/2cd42d156243067a7663bb0bf0156e5d87188c2f parameter order upgrade bug] that breaks VTK examples.<br />
* [TE] Matplotlib and sklearn are exportable using module-to-script api<br />
* [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.<br />
<br />
'''Items to Discuss'''<br />
* Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!<br />
* Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)<br />
* [TE] Wrap-up notes<br />
<br />
=== August 10, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Python Wrapper and scripting<br />
** All changes merged into python-wrapper branch<br />
*** Tests pass<br />
*** Still need to update sub-branches<br />
** Minor fixes<br />
*** vtkPlot3DReader upgrade bug<br />
*** Added tuple exporting<br />
** Most of vtk examples can be exported [https://gist.github.com/rexissimus/5ac7374cde8525d84f9443f019d234e2 example]<br />
*** Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary<br />
**** Could be a bug in the old vistrail, but not sure<br />
* [DK] Subworkflows and uuids<br />
<br />
=== August 3, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Wrapper patches now stored as [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/vtk_patch.py python code]<br />
** Connection variables now lowercase<br />
** Can we use code as output variables?<br />
*** E.g. "vtk_dataset_mapper.GetOutput(0)"<br />
*** Could mutate output when output method is only supposed to be called once<br />
*** Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)<br />
<br />
=== July 27, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] DB bundle serialization implemented<br />
<br />
'''Items to Discuss'''<br />
* [TE] Finished using patch in script [https://github.com/VisTrails/VisTrails/tree/python-wrapper-script-patches python-wrapper-script-patches]<br />
** locale?<br />
** [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d new patching example] vs [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 old patching example]<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/parse.py#L135-L273 Example of new patches] <br />
** Legacy '.vtkInstance' is now added as a patch "self.vtkInstance=self" (see new patching example)<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/api.py New api for executing standalone modules]<br />
*** Accessing module was disabled for core/api, but a different api for this is probably better anyway.<br />
*** api.Package now gives [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d#file-new_patching-py-L26 directly callable module compute functions]<br />
*** Only works on modules not using interpreter functionality ('.interpreter', etc.)<br />
**** for zip file modules and others that need temp files and use the interpreter for this, need to modify the module, but could work around with python libraries for managing temp files<br />
** Fixed more renaming issues - [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/scripts.py#L63-L64 function argument names should not be replaced]<br />
** Still need to do something nicer for the patch formatting<br />
*** patch keys currently contain non-python-variable characters but this can be fixed<br />
*** Need to replace string template with python variables that is replacable using RedBaron<br />
**** Patches could then look like:<br />
<br />
def vtkInstance():<br />
self.vtkInstance = self<br />
<br />
=== July 20, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] New patch method for python-wrapper [https://gist.github.com/rexissimus/009b303e4cadcd4b8f2fec6cf848cb2b example]<br />
** A patch is a string template with self/input/output variables for patching a class method <br />
** Patches are stored in xml specification<br />
** Patches are assigned to methods in ClassSpec's<br />
** Used both for execution in vistrails and when exporting as script<br />
*** Executed with "exec" when run in vistrails<br />
*** Turns patches into natural calls in script<br />
** Wrapping VTK library no longer needed<br />
*** Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)<br />
** Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)<br />
** Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)<br />
** TODO's<br />
*** Add to export_as_script<br />
*** Patch Modules that represent Python functions as well<br />
<br />
* [vistrails-users] Read data from SQL example?<br />
<br />
=== July 13, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] bundles on sql-alchemy branch<br />
** Bundle, BundleMapping, and BundleSerializer<br />
** New BaseSerializers to deal with versions of directory, zip, db serializers<br />
** Can register the BundleSerializers with the base serializers<br />
** Translation between bundle versions<br />
** Bundle.add_object "just works" if an appropriate mapping was registered<br />
<br />
'''Items to Discuss'''<br />
* [TE] Simplified running module using API for scripting [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 example]<br />
** API Module can be executed using '''compute''' method<br />
*** Takes list of outputs to compute + inputs as keyword arguments<br />
* Status of general cell synchronization (vistrails-users)<br />
<br />
=== July 6, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] Working on bundles<br />
<br />
'''Items to Discuss'''<br />
* [TE] Using vistrails API in scripting [https://gist.github.com/rexissimus/034d9c4dedc3299a079e84981dedd8c7 example 1 (API with VTK)] [https://gist.github.com/rexissimus/a4a9ca6c08d4058c64967d2b35415fd5 example 2 (DownloadFile only)]<br />
** Call module in script using vistrails API<br />
** Can be used when no export method is defined.<br />
* [TE] [https://github.com/VisTrails/VisTrails/commit/c76c109349769fee1280f85c22678380a2f9cd39 Server news bug] Critical?<br />
**[RR] No, unreleased<br />
* [DK] New bundle organization<br />
** BundleMapping contains individual mappings from raw objects to BundleObj<br />
** Must be separate in order to have in-memory, separate from serializer settings<br />
<br />
=== June 29, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as python script<br />
** Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) [https://gist.github.com/rexissimus/16caf5200802e150296f62a0c7844c5a example]<br />
** Preludes (imports) still need improvements<br />
** Works with vtkInteractionHandler (clipping plane)<br />
** Resulting clipping plane is incorrect? (supposed to hide one side of the plane)<br />
** vtkRendererOutput Uses VTKRenderWindow as output<br />
*** Should support different output modes?<br />
*** Does not work with multiple outputs (Next cell is displayed when current is closed)<br />
*** Uses wrapped vtkInstance (as vvtk)<br />
**** Create new patch system with code that can be used both in vistrails and in a python script?<br />
<br />
=== June 22, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fix for upgraded analogies (PR [https://github.com/VisTrails/VisTrails/pull/1177 #1177])<br />
** Uses module location to find module remap (The best we can do right now?)<br />
** Also fixes remapped port names for connections to existing modules <br />
** May still fail on complex upgrades like adding a function on an upgraded module that has a different port name<br />
<br />
* [TE] Working on exporting wrapped libraries (VTK)<br />
<br />
=== June 15, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Added depth logic to python script exporter ([https://github.com/VisTrails/VisTrails/commit/50f220e4f6abe5a7bb8689f73f456f43df2dbc12 full commit])<br />
** Fully implements connection merging and list looping<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Making analogy work on upgraded workflows ([https://github.com/VisTrails/VisTrails/issues/1175 #1175])<br />
** Analogy only works We need to use the non-upgraded action chain<br />
** But pipelines need to be valid (matcher uses module portspecs)<br />
** Tried to get upgrade remap from upgrade action, but this seems impossible<br />
** Use module location instead?<br />
*** Location should be identical for one-to-one module upgrades<br />
<br />
=== June 1, 2016 ===<br />
<br />
'''Updates'''<br />
* VisTrails 2.2.4 announced (finally)<br />
** 5 survey replies<br />
** 47 downloads last week<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as script (python-wrapper)<br />
** to_python_script<br />
*** needs the pipeline (or at least the connections)<br />
**** Should not generate code for any possible VTK method (exporter detects which ports is used in the source)<br />
**** VTK needs to know connection order<br />
**** Adding functions directly to the script results in nicer code<br />
** Add support for multiple input connections<br />
** Add support for list looping<br />
<br />
=== May 25, 2016 ===<br />
<br />
'''Updates'''<br />
* Mailing-list fixed<br />
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])<br />
<br />
'''Items to Discuss'''<br />
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)<br />
* [TE] Union ports<br />
** Show union ports in pipeline (DEMO)<br />
<br />
=== May 18, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] mailing list still down - Julio working on it<br />
* [TE] Improved union port<br />
** New schema 1.0.5 adds portitem.union string attribute<br />
** Works with pipeline view, parameter exploration view, and mashup view<br />
** Should work well for matplotlib<br />
** Use with VTK ports?<br />
*** It uses [item_1,item_2] style ports usually with different tuple length.<br />
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]<br />
<br />
=== May 11, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?<br />
*** Shows single port for all ports with same sort_key<br />
*** Select specific type when adding/showing<br />
*** Only one type should be visible on the module at any time<br />
*** Making tuple parameters union types is probably too complicated<br />
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly) <br />
<br />
* [TE] Feature suggestion: Let connections connect to functions/parameters<br />
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters<br />
** Use separate views for ports and functions (like before)<br />
*** Allow rearranging of functions<br />
** Pros:<br />
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.<br />
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.<br />
**** Makes List module simpler<br />
*** Connect to parameter in tuple<br />
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds. <br />
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.<br />
** Cons:<br />
*** Interface more complicated and crowded<br />
*** More difficult for users to grasp?<br />
*** No immediate use case<br />
*** Makes no union type for tuple parameters a bigger problem.<br />
<br />
* [TE] Publish user survey?<br />
** No response from vistrails release yet<br />
** Not announced on vistrails-users yet.<br />
<br />
=== May 4, 2016 ===<br />
'''Updates'''<br />
* VisTrails 2.2.4 released<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
<br />
=== April 27, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Wrapping Bokeh<br />
** New method type "operation"<br />
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output<br />
**** Glyph sets function name and arguments as output<br />
***** Not serializable, but should always be computed together with the subject of the operation.<br />
**** Figure applies Glyph operation on itself<br />
*** Useful for matplotlib?<br />
**** matplotlib currently find actors by searching upstream?<br />
**** But this is more difficult since matplotlib is not function-based.<br />
** New "union" port type<br />
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value<br />
**** Uses only first value in tuple<br />
**** Bokeh have many properties that support (list, string, Float)<br />
*** Does not work with depth=1 types (e.g. Float of depth 1)<br />
**** Turn Depth 1 types with known type into lists?<br />
<br />
* [TE] New bugfix release<br />
** Finalize survey<br />
<br />
=== April 20, 2016 ===<br />
'''Updates'''<br />
* Merged upgrade fixes<br />
** Now has infinite loop test<br />
** loop test now correctly loops 5 times<br />
*** remap_module used wrong (current) package version when creating ModuleDescriptor<br />
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])<br />
<br />
'''Items to Discuss'''<br />
* New bugfix release<br />
** Merge usagestats<br />
** Need to finalize survey<br />
<br />
=== April 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?<br />
<br />
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])<br />
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]<br />
** Cherry-pick test to preserve source?<br />
** Need maxPipelineFixAttempts option?<br />
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.<br />
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?<br />
*** Would need to be moved into handle_invalid_pipeline<br />
<br />
=== April 6, 2016 ===<br />
'''Updates'''<br />
* [TE] Query View improvements<br />
** Done except for displaying "Version Info" and #1164<br />
* [RR] Working on finishing new interpreter (finally!)<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Try latest upgrade first, then previous.<br />
** If all fail, select latest upgrade?<br />
<br />
=== March 30, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Query View improvements<br />
** Unified do_version_switch and get_upgrades as validate_version()<br />
** Problems with query view using its own controller<br />
*** Fixed module info not being displayed for query results<br />
*** Still no version info for query results<br />
*** Still an issue with version query view not being updated<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== Older meetings ==<br />
* [[Development/2015]]<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=12581Development2016-10-19T16:17:53Z<p>Remi: /* October 19, 2016 */</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2016 ==<br />
=== November ??, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* New version 2.2-2 of Debian package, still [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=796994 wrong copyright], and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=748692 broken bundle installation]<br />
** Finally replied, says copyright issue will be fixed soon, working on including VisTrails 2.2.4 for Debian Stretch<br />
<br />
'''Items to Discuss'''<br />
<br />
=== October 5, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
<br />
* working toward a 3.0 release; [https://github.com/VisTrails/VisTrails/projects/1 project board]<br />
** unique ids<br />
** bundles<br />
** GUI fixes<br />
** wrapping updates<br />
** python3<br />
** interpreter?<br />
<br />
* command-line input parameters to provenance<br />
** [DK] we don't create a persisted workflow from those input params, i think<br />
** [DK] these are already defined as aliases so we treat them differently form a normal input param<br />
** [DK] still have the potential problem of allowing a user to view the workflow as it was run with a given set of aliases<br />
** Do the vistrails variables values get stored in provenance? (what if values change?)<br />
<br />
=== September 14, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] [http://aosabook.org/en/vistrails.html VisTrails] in [http://aosabook.org/ The Architecture of Open Source Applications] Vol I (2012)<br />
** Did we know/contribute to this?<br />
*** [DK] Yes. We wrote the text for the VisTrails chapter<br />
** Drafted response<br />
*** [DK] Looks good, will forward.<br />
<br />
=== August 31, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Went over vgc.poly.edu/vistrails.org config:<br />
** Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)<br />
** Fixed usersguide scripts (won't disappear again)<br />
** Moved some websites over to HTTPS<br />
** [https://docs.google.com/document/d/18YZewjR2YY9cQ9pzdA-nU9jbu2L7jYcPlDi3iaPadwA/edit Websites]<br />
* [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs ([http://vistrails.readthedocs.io/ currently there]) and users' guide under the same RTD "project"<br />
* [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that<br />
<br />
'''Items to Discuss'''<br />
* [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/<br />
** Get rid of more services? (svn, trac, ...)<br />
* [RR] Notebook execution works. How do we support more languages (than Python)?<br />
** Need: detection, metadata reader, code to input values<br />
* [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?<br />
<br />
=== August 17, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Fixed [https://github.com/VisTrails/VisTrails/commit/2cd42d156243067a7663bb0bf0156e5d87188c2f parameter order upgrade bug] that breaks VTK examples.<br />
* [TE] Matplotlib and sklearn are exportable using module-to-script api<br />
* [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.<br />
<br />
'''Items to Discuss'''<br />
* Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!<br />
* Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)<br />
* [TE] Wrap-up notes<br />
<br />
=== August 10, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Python Wrapper and scripting<br />
** All changes merged into python-wrapper branch<br />
*** Tests pass<br />
*** Still need to update sub-branches<br />
** Minor fixes<br />
*** vtkPlot3DReader upgrade bug<br />
*** Added tuple exporting<br />
** Most of vtk examples can be exported [https://gist.github.com/rexissimus/5ac7374cde8525d84f9443f019d234e2 example]<br />
*** Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary<br />
**** Could be a bug in the old vistrail, but not sure<br />
* [DK] Subworkflows and uuids<br />
<br />
=== August 3, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Wrapper patches now stored as [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/vtk_patch.py python code]<br />
** Connection variables now lowercase<br />
** Can we use code as output variables?<br />
*** E.g. "vtk_dataset_mapper.GetOutput(0)"<br />
*** Could mutate output when output method is only supposed to be called once<br />
*** Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)<br />
<br />
=== July 27, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] DB bundle serialization implemented<br />
<br />
'''Items to Discuss'''<br />
* [TE] Finished using patch in script [https://github.com/VisTrails/VisTrails/tree/python-wrapper-script-patches python-wrapper-script-patches]<br />
** locale?<br />
** [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d new patching example] vs [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 old patching example]<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/parse.py#L135-L273 Example of new patches] <br />
** Legacy '.vtkInstance' is now added as a patch "self.vtkInstance=self" (see new patching example)<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/api.py New api for executing standalone modules]<br />
*** Accessing module was disabled for core/api, but a different api for this is probably better anyway.<br />
*** api.Package now gives [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d#file-new_patching-py-L26 directly callable module compute functions]<br />
*** Only works on modules not using interpreter functionality ('.interpreter', etc.)<br />
**** for zip file modules and others that need temp files and use the interpreter for this, need to modify the module, but could work around with python libraries for managing temp files<br />
** Fixed more renaming issues - [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/scripts.py#L63-L64 function argument names should not be replaced]<br />
** Still need to do something nicer for the patch formatting<br />
*** patch keys currently contain non-python-variable characters but this can be fixed<br />
*** Need to replace string template with python variables that is replacable using RedBaron<br />
**** Patches could then look like:<br />
<br />
def vtkInstance():<br />
self.vtkInstance = self<br />
<br />
=== July 20, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] New patch method for python-wrapper [https://gist.github.com/rexissimus/009b303e4cadcd4b8f2fec6cf848cb2b example]<br />
** A patch is a string template with self/input/output variables for patching a class method <br />
** Patches are stored in xml specification<br />
** Patches are assigned to methods in ClassSpec's<br />
** Used both for execution in vistrails and when exporting as script<br />
*** Executed with "exec" when run in vistrails<br />
*** Turns patches into natural calls in script<br />
** Wrapping VTK library no longer needed<br />
*** Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)<br />
** Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)<br />
** Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)<br />
** TODO's<br />
*** Add to export_as_script<br />
*** Patch Modules that represent Python functions as well<br />
<br />
* [vistrails-users] Read data from SQL example?<br />
<br />
=== July 13, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] bundles on sql-alchemy branch<br />
** Bundle, BundleMapping, and BundleSerializer<br />
** New BaseSerializers to deal with versions of directory, zip, db serializers<br />
** Can register the BundleSerializers with the base serializers<br />
** Translation between bundle versions<br />
** Bundle.add_object "just works" if an appropriate mapping was registered<br />
<br />
'''Items to Discuss'''<br />
* [TE] Simplified running module using API for scripting [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 example]<br />
** API Module can be executed using '''compute''' method<br />
*** Takes list of outputs to compute + inputs as keyword arguments<br />
* Status of general cell synchronization (vistrails-users)<br />
<br />
=== July 6, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] Working on bundles<br />
<br />
'''Items to Discuss'''<br />
* [TE] Using vistrails API in scripting [https://gist.github.com/rexissimus/034d9c4dedc3299a079e84981dedd8c7 example 1 (API with VTK)] [https://gist.github.com/rexissimus/a4a9ca6c08d4058c64967d2b35415fd5 example 2 (DownloadFile only)]<br />
** Call module in script using vistrails API<br />
** Can be used when no export method is defined.<br />
* [TE] [https://github.com/VisTrails/VisTrails/commit/c76c109349769fee1280f85c22678380a2f9cd39 Server news bug] Critical?<br />
**[RR] No, unreleased<br />
* [DK] New bundle organization<br />
** BundleMapping contains individual mappings from raw objects to BundleObj<br />
** Must be separate in order to have in-memory, separate from serializer settings<br />
<br />
=== June 29, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as python script<br />
** Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) [https://gist.github.com/rexissimus/16caf5200802e150296f62a0c7844c5a example]<br />
** Preludes (imports) still need improvements<br />
** Works with vtkInteractionHandler (clipping plane)<br />
** Resulting clipping plane is incorrect? (supposed to hide one side of the plane)<br />
** vtkRendererOutput Uses VTKRenderWindow as output<br />
*** Should support different output modes?<br />
*** Does not work with multiple outputs (Next cell is displayed when current is closed)<br />
*** Uses wrapped vtkInstance (as vvtk)<br />
**** Create new patch system with code that can be used both in vistrails and in a python script?<br />
<br />
=== June 22, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fix for upgraded analogies (PR [https://github.com/VisTrails/VisTrails/pull/1177 #1177])<br />
** Uses module location to find module remap (The best we can do right now?)<br />
** Also fixes remapped port names for connections to existing modules <br />
** May still fail on complex upgrades like adding a function on an upgraded module that has a different port name<br />
<br />
* [TE] Working on exporting wrapped libraries (VTK)<br />
<br />
=== June 15, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Added depth logic to python script exporter ([https://github.com/VisTrails/VisTrails/commit/50f220e4f6abe5a7bb8689f73f456f43df2dbc12 full commit])<br />
** Fully implements connection merging and list looping<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Making analogy work on upgraded workflows ([https://github.com/VisTrails/VisTrails/issues/1175 #1175])<br />
** Analogy only works We need to use the non-upgraded action chain<br />
** But pipelines need to be valid (matcher uses module portspecs)<br />
** Tried to get upgrade remap from upgrade action, but this seems impossible<br />
** Use module location instead?<br />
*** Location should be identical for one-to-one module upgrades<br />
<br />
=== June 1, 2016 ===<br />
<br />
'''Updates'''<br />
* VisTrails 2.2.4 announced (finally)<br />
** 5 survey replies<br />
** 47 downloads last week<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as script (python-wrapper)<br />
** to_python_script<br />
*** needs the pipeline (or at least the connections)<br />
**** Should not generate code for any possible VTK method (exporter detects which ports is used in the source)<br />
**** VTK needs to know connection order<br />
**** Adding functions directly to the script results in nicer code<br />
** Add support for multiple input connections<br />
** Add support for list looping<br />
<br />
=== May 25, 2016 ===<br />
<br />
'''Updates'''<br />
* Mailing-list fixed<br />
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])<br />
<br />
'''Items to Discuss'''<br />
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)<br />
* [TE] Union ports<br />
** Show union ports in pipeline (DEMO)<br />
<br />
=== May 18, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] mailing list still down - Julio working on it<br />
* [TE] Improved union port<br />
** New schema 1.0.5 adds portitem.union string attribute<br />
** Works with pipeline view, parameter exploration view, and mashup view<br />
** Should work well for matplotlib<br />
** Use with VTK ports?<br />
*** It uses [item_1,item_2] style ports usually with different tuple length.<br />
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]<br />
<br />
=== May 11, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?<br />
*** Shows single port for all ports with same sort_key<br />
*** Select specific type when adding/showing<br />
*** Only one type should be visible on the module at any time<br />
*** Making tuple parameters union types is probably too complicated<br />
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly) <br />
<br />
* [TE] Feature suggestion: Let connections connect to functions/parameters<br />
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters<br />
** Use separate views for ports and functions (like before)<br />
*** Allow rearranging of functions<br />
** Pros:<br />
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.<br />
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.<br />
**** Makes List module simpler<br />
*** Connect to parameter in tuple<br />
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds. <br />
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.<br />
** Cons:<br />
*** Interface more complicated and crowded<br />
*** More difficult for users to grasp?<br />
*** No immediate use case<br />
*** Makes no union type for tuple parameters a bigger problem.<br />
<br />
* [TE] Publish user survey?<br />
** No response from vistrails release yet<br />
** Not announced on vistrails-users yet.<br />
<br />
=== May 4, 2016 ===<br />
'''Updates'''<br />
* VisTrails 2.2.4 released<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
<br />
=== April 27, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Wrapping Bokeh<br />
** New method type "operation"<br />
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output<br />
**** Glyph sets function name and arguments as output<br />
***** Not serializable, but should always be computed together with the subject of the operation.<br />
**** Figure applies Glyph operation on itself<br />
*** Useful for matplotlib?<br />
**** matplotlib currently find actors by searching upstream?<br />
**** But this is more difficult since matplotlib is not function-based.<br />
** New "union" port type<br />
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value<br />
**** Uses only first value in tuple<br />
**** Bokeh have many properties that support (list, string, Float)<br />
*** Does not work with depth=1 types (e.g. Float of depth 1)<br />
**** Turn Depth 1 types with known type into lists?<br />
<br />
* [TE] New bugfix release<br />
** Finalize survey<br />
<br />
=== April 20, 2016 ===<br />
'''Updates'''<br />
* Merged upgrade fixes<br />
** Now has infinite loop test<br />
** loop test now correctly loops 5 times<br />
*** remap_module used wrong (current) package version when creating ModuleDescriptor<br />
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])<br />
<br />
'''Items to Discuss'''<br />
* New bugfix release<br />
** Merge usagestats<br />
** Need to finalize survey<br />
<br />
=== April 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?<br />
<br />
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])<br />
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]<br />
** Cherry-pick test to preserve source?<br />
** Need maxPipelineFixAttempts option?<br />
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.<br />
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?<br />
*** Would need to be moved into handle_invalid_pipeline<br />
<br />
=== April 6, 2016 ===<br />
'''Updates'''<br />
* [TE] Query View improvements<br />
** Done except for displaying "Version Info" and #1164<br />
* [RR] Working on finishing new interpreter (finally!)<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Try latest upgrade first, then previous.<br />
** If all fail, select latest upgrade?<br />
<br />
=== March 30, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Query View improvements<br />
** Unified do_version_switch and get_upgrades as validate_version()<br />
** Problems with query view using its own controller<br />
*** Fixed module info not being displayed for query results<br />
*** Still no version info for query results<br />
*** Still an issue with version query view not being updated<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== Older meetings ==<br />
* [[Development/2015]]<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=12580Development2016-10-19T14:54:16Z<p>Remi: /* October 19, 2016 */ Got reply from Debian developer</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2016 ==<br />
=== October 19, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* New version 2.2-2 of Debian package, still [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=796994 wrong copyright], and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=748692 broken bundle installation]<br />
** Finally replied, says copyright issue will be fixed soon, working on including VisTrails 2.2.4 for Debian Stretch<br />
<br />
'''Items to Discuss'''<br />
<br />
=== October 5, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
<br />
* working toward a 3.0 release; [https://github.com/VisTrails/VisTrails/projects/1 project board]<br />
** unique ids<br />
** bundles<br />
** GUI fixes<br />
** wrapping updates<br />
** python3<br />
** interpreter?<br />
<br />
* command-line input parameters to provenance<br />
** [DK] we don't create a persisted workflow from those input params, i think<br />
** [DK] these are already defined as aliases so we treat them differently form a normal input param<br />
** [DK] still have the potential problem of allowing a user to view the workflow as it was run with a given set of aliases<br />
** Do the vistrails variables values get stored in provenance? (what if values change?)<br />
<br />
=== September 14, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] [http://aosabook.org/en/vistrails.html VisTrails] in [http://aosabook.org/ The Architecture of Open Source Applications] Vol I (2012)<br />
** Did we know/contribute to this?<br />
*** [DK] Yes. We wrote the text for the VisTrails chapter<br />
** Drafted response<br />
*** [DK] Looks good, will forward.<br />
<br />
=== August 31, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Went over vgc.poly.edu/vistrails.org config:<br />
** Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)<br />
** Fixed usersguide scripts (won't disappear again)<br />
** Moved some websites over to HTTPS<br />
** [https://docs.google.com/document/d/18YZewjR2YY9cQ9pzdA-nU9jbu2L7jYcPlDi3iaPadwA/edit Websites]<br />
* [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs ([http://vistrails.readthedocs.io/ currently there]) and users' guide under the same RTD "project"<br />
* [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that<br />
<br />
'''Items to Discuss'''<br />
* [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/<br />
** Get rid of more services? (svn, trac, ...)<br />
* [RR] Notebook execution works. How do we support more languages (than Python)?<br />
** Need: detection, metadata reader, code to input values<br />
* [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?<br />
<br />
=== August 17, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Fixed [https://github.com/VisTrails/VisTrails/commit/2cd42d156243067a7663bb0bf0156e5d87188c2f parameter order upgrade bug] that breaks VTK examples.<br />
* [TE] Matplotlib and sklearn are exportable using module-to-script api<br />
* [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.<br />
<br />
'''Items to Discuss'''<br />
* Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!<br />
* Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)<br />
* [TE] Wrap-up notes<br />
<br />
=== August 10, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Python Wrapper and scripting<br />
** All changes merged into python-wrapper branch<br />
*** Tests pass<br />
*** Still need to update sub-branches<br />
** Minor fixes<br />
*** vtkPlot3DReader upgrade bug<br />
*** Added tuple exporting<br />
** Most of vtk examples can be exported [https://gist.github.com/rexissimus/5ac7374cde8525d84f9443f019d234e2 example]<br />
*** Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary<br />
**** Could be a bug in the old vistrail, but not sure<br />
* [DK] Subworkflows and uuids<br />
<br />
=== August 3, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Wrapper patches now stored as [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/vtk_patch.py python code]<br />
** Connection variables now lowercase<br />
** Can we use code as output variables?<br />
*** E.g. "vtk_dataset_mapper.GetOutput(0)"<br />
*** Could mutate output when output method is only supposed to be called once<br />
*** Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)<br />
<br />
=== July 27, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] DB bundle serialization implemented<br />
<br />
'''Items to Discuss'''<br />
* [TE] Finished using patch in script [https://github.com/VisTrails/VisTrails/tree/python-wrapper-script-patches python-wrapper-script-patches]<br />
** locale?<br />
** [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d new patching example] vs [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 old patching example]<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/parse.py#L135-L273 Example of new patches] <br />
** Legacy '.vtkInstance' is now added as a patch "self.vtkInstance=self" (see new patching example)<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/api.py New api for executing standalone modules]<br />
*** Accessing module was disabled for core/api, but a different api for this is probably better anyway.<br />
*** api.Package now gives [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d#file-new_patching-py-L26 directly callable module compute functions]<br />
*** Only works on modules not using interpreter functionality ('.interpreter', etc.)<br />
**** for zip file modules and others that need temp files and use the interpreter for this, need to modify the module, but could work around with python libraries for managing temp files<br />
** Fixed more renaming issues - [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/scripts.py#L63-L64 function argument names should not be replaced]<br />
** Still need to do something nicer for the patch formatting<br />
*** patch keys currently contain non-python-variable characters but this can be fixed<br />
*** Need to replace string template with python variables that is replacable using RedBaron<br />
**** Patches could then look like:<br />
<br />
def vtkInstance():<br />
self.vtkInstance = self<br />
<br />
=== July 20, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] New patch method for python-wrapper [https://gist.github.com/rexissimus/009b303e4cadcd4b8f2fec6cf848cb2b example]<br />
** A patch is a string template with self/input/output variables for patching a class method <br />
** Patches are stored in xml specification<br />
** Patches are assigned to methods in ClassSpec's<br />
** Used both for execution in vistrails and when exporting as script<br />
*** Executed with "exec" when run in vistrails<br />
*** Turns patches into natural calls in script<br />
** Wrapping VTK library no longer needed<br />
*** Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)<br />
** Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)<br />
** Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)<br />
** TODO's<br />
*** Add to export_as_script<br />
*** Patch Modules that represent Python functions as well<br />
<br />
* [vistrails-users] Read data from SQL example?<br />
<br />
=== July 13, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] bundles on sql-alchemy branch<br />
** Bundle, BundleMapping, and BundleSerializer<br />
** New BaseSerializers to deal with versions of directory, zip, db serializers<br />
** Can register the BundleSerializers with the base serializers<br />
** Translation between bundle versions<br />
** Bundle.add_object "just works" if an appropriate mapping was registered<br />
<br />
'''Items to Discuss'''<br />
* [TE] Simplified running module using API for scripting [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 example]<br />
** API Module can be executed using '''compute''' method<br />
*** Takes list of outputs to compute + inputs as keyword arguments<br />
* Status of general cell synchronization (vistrails-users)<br />
<br />
=== July 6, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] Working on bundles<br />
<br />
'''Items to Discuss'''<br />
* [TE] Using vistrails API in scripting [https://gist.github.com/rexissimus/034d9c4dedc3299a079e84981dedd8c7 example 1 (API with VTK)] [https://gist.github.com/rexissimus/a4a9ca6c08d4058c64967d2b35415fd5 example 2 (DownloadFile only)]<br />
** Call module in script using vistrails API<br />
** Can be used when no export method is defined.<br />
* [TE] [https://github.com/VisTrails/VisTrails/commit/c76c109349769fee1280f85c22678380a2f9cd39 Server news bug] Critical?<br />
**[RR] No, unreleased<br />
* [DK] New bundle organization<br />
** BundleMapping contains individual mappings from raw objects to BundleObj<br />
** Must be separate in order to have in-memory, separate from serializer settings<br />
<br />
=== June 29, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as python script<br />
** Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) [https://gist.github.com/rexissimus/16caf5200802e150296f62a0c7844c5a example]<br />
** Preludes (imports) still need improvements<br />
** Works with vtkInteractionHandler (clipping plane)<br />
** Resulting clipping plane is incorrect? (supposed to hide one side of the plane)<br />
** vtkRendererOutput Uses VTKRenderWindow as output<br />
*** Should support different output modes?<br />
*** Does not work with multiple outputs (Next cell is displayed when current is closed)<br />
*** Uses wrapped vtkInstance (as vvtk)<br />
**** Create new patch system with code that can be used both in vistrails and in a python script?<br />
<br />
=== June 22, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fix for upgraded analogies (PR [https://github.com/VisTrails/VisTrails/pull/1177 #1177])<br />
** Uses module location to find module remap (The best we can do right now?)<br />
** Also fixes remapped port names for connections to existing modules <br />
** May still fail on complex upgrades like adding a function on an upgraded module that has a different port name<br />
<br />
* [TE] Working on exporting wrapped libraries (VTK)<br />
<br />
=== June 15, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Added depth logic to python script exporter ([https://github.com/VisTrails/VisTrails/commit/50f220e4f6abe5a7bb8689f73f456f43df2dbc12 full commit])<br />
** Fully implements connection merging and list looping<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Making analogy work on upgraded workflows ([https://github.com/VisTrails/VisTrails/issues/1175 #1175])<br />
** Analogy only works We need to use the non-upgraded action chain<br />
** But pipelines need to be valid (matcher uses module portspecs)<br />
** Tried to get upgrade remap from upgrade action, but this seems impossible<br />
** Use module location instead?<br />
*** Location should be identical for one-to-one module upgrades<br />
<br />
=== June 1, 2016 ===<br />
<br />
'''Updates'''<br />
* VisTrails 2.2.4 announced (finally)<br />
** 5 survey replies<br />
** 47 downloads last week<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as script (python-wrapper)<br />
** to_python_script<br />
*** needs the pipeline (or at least the connections)<br />
**** Should not generate code for any possible VTK method (exporter detects which ports is used in the source)<br />
**** VTK needs to know connection order<br />
**** Adding functions directly to the script results in nicer code<br />
** Add support for multiple input connections<br />
** Add support for list looping<br />
<br />
=== May 25, 2016 ===<br />
<br />
'''Updates'''<br />
* Mailing-list fixed<br />
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])<br />
<br />
'''Items to Discuss'''<br />
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)<br />
* [TE] Union ports<br />
** Show union ports in pipeline (DEMO)<br />
<br />
=== May 18, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] mailing list still down - Julio working on it<br />
* [TE] Improved union port<br />
** New schema 1.0.5 adds portitem.union string attribute<br />
** Works with pipeline view, parameter exploration view, and mashup view<br />
** Should work well for matplotlib<br />
** Use with VTK ports?<br />
*** It uses [item_1,item_2] style ports usually with different tuple length.<br />
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]<br />
<br />
=== May 11, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?<br />
*** Shows single port for all ports with same sort_key<br />
*** Select specific type when adding/showing<br />
*** Only one type should be visible on the module at any time<br />
*** Making tuple parameters union types is probably too complicated<br />
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly) <br />
<br />
* [TE] Feature suggestion: Let connections connect to functions/parameters<br />
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters<br />
** Use separate views for ports and functions (like before)<br />
*** Allow rearranging of functions<br />
** Pros:<br />
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.<br />
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.<br />
**** Makes List module simpler<br />
*** Connect to parameter in tuple<br />
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds. <br />
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.<br />
** Cons:<br />
*** Interface more complicated and crowded<br />
*** More difficult for users to grasp?<br />
*** No immediate use case<br />
*** Makes no union type for tuple parameters a bigger problem.<br />
<br />
* [TE] Publish user survey?<br />
** No response from vistrails release yet<br />
** Not announced on vistrails-users yet.<br />
<br />
=== May 4, 2016 ===<br />
'''Updates'''<br />
* VisTrails 2.2.4 released<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
<br />
=== April 27, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Wrapping Bokeh<br />
** New method type "operation"<br />
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output<br />
**** Glyph sets function name and arguments as output<br />
***** Not serializable, but should always be computed together with the subject of the operation.<br />
**** Figure applies Glyph operation on itself<br />
*** Useful for matplotlib?<br />
**** matplotlib currently find actors by searching upstream?<br />
**** But this is more difficult since matplotlib is not function-based.<br />
** New "union" port type<br />
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value<br />
**** Uses only first value in tuple<br />
**** Bokeh have many properties that support (list, string, Float)<br />
*** Does not work with depth=1 types (e.g. Float of depth 1)<br />
**** Turn Depth 1 types with known type into lists?<br />
<br />
* [TE] New bugfix release<br />
** Finalize survey<br />
<br />
=== April 20, 2016 ===<br />
'''Updates'''<br />
* Merged upgrade fixes<br />
** Now has infinite loop test<br />
** loop test now correctly loops 5 times<br />
*** remap_module used wrong (current) package version when creating ModuleDescriptor<br />
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])<br />
<br />
'''Items to Discuss'''<br />
* New bugfix release<br />
** Merge usagestats<br />
** Need to finalize survey<br />
<br />
=== April 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?<br />
<br />
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])<br />
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]<br />
** Cherry-pick test to preserve source?<br />
** Need maxPipelineFixAttempts option?<br />
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.<br />
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?<br />
*** Would need to be moved into handle_invalid_pipeline<br />
<br />
=== April 6, 2016 ===<br />
'''Updates'''<br />
* [TE] Query View improvements<br />
** Done except for displaying "Version Info" and #1164<br />
* [RR] Working on finishing new interpreter (finally!)<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Try latest upgrade first, then previous.<br />
** If all fail, select latest upgrade?<br />
<br />
=== March 30, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Query View improvements<br />
** Unified do_version_switch and get_upgrades as validate_version()<br />
** Problems with query view using its own controller<br />
*** Fixed module info not being displayed for query results<br />
*** Still no version info for query results<br />
*** Still an issue with version query view not being updated<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== Older meetings ==<br />
* [[Development/2015]]<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=12579Development2016-10-19T14:53:06Z<p>Remi: /* October 19, 2016 */ Fix markup</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2016 ==<br />
=== October 19, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* New version 2.2-2 of Debian package, still [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=796994 wrong copyright], and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=748692 broken bundle installation]<br />
<br />
'''Items to Discuss'''<br />
<br />
=== October 5, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
<br />
* working toward a 3.0 release; [https://github.com/VisTrails/VisTrails/projects/1 project board]<br />
** unique ids<br />
** bundles<br />
** GUI fixes<br />
** wrapping updates<br />
** python3<br />
** interpreter?<br />
<br />
* command-line input parameters to provenance<br />
** [DK] we don't create a persisted workflow from those input params, i think<br />
** [DK] these are already defined as aliases so we treat them differently form a normal input param<br />
** [DK] still have the potential problem of allowing a user to view the workflow as it was run with a given set of aliases<br />
** Do the vistrails variables values get stored in provenance? (what if values change?)<br />
<br />
=== September 14, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] [http://aosabook.org/en/vistrails.html VisTrails] in [http://aosabook.org/ The Architecture of Open Source Applications] Vol I (2012)<br />
** Did we know/contribute to this?<br />
*** [DK] Yes. We wrote the text for the VisTrails chapter<br />
** Drafted response<br />
*** [DK] Looks good, will forward.<br />
<br />
=== August 31, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Went over vgc.poly.edu/vistrails.org config:<br />
** Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)<br />
** Fixed usersguide scripts (won't disappear again)<br />
** Moved some websites over to HTTPS<br />
** [https://docs.google.com/document/d/18YZewjR2YY9cQ9pzdA-nU9jbu2L7jYcPlDi3iaPadwA/edit Websites]<br />
* [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs ([http://vistrails.readthedocs.io/ currently there]) and users' guide under the same RTD "project"<br />
* [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that<br />
<br />
'''Items to Discuss'''<br />
* [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/<br />
** Get rid of more services? (svn, trac, ...)<br />
* [RR] Notebook execution works. How do we support more languages (than Python)?<br />
** Need: detection, metadata reader, code to input values<br />
* [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?<br />
<br />
=== August 17, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Fixed [https://github.com/VisTrails/VisTrails/commit/2cd42d156243067a7663bb0bf0156e5d87188c2f parameter order upgrade bug] that breaks VTK examples.<br />
* [TE] Matplotlib and sklearn are exportable using module-to-script api<br />
* [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.<br />
<br />
'''Items to Discuss'''<br />
* Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!<br />
* Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)<br />
* [TE] Wrap-up notes<br />
<br />
=== August 10, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Python Wrapper and scripting<br />
** All changes merged into python-wrapper branch<br />
*** Tests pass<br />
*** Still need to update sub-branches<br />
** Minor fixes<br />
*** vtkPlot3DReader upgrade bug<br />
*** Added tuple exporting<br />
** Most of vtk examples can be exported [https://gist.github.com/rexissimus/5ac7374cde8525d84f9443f019d234e2 example]<br />
*** Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary<br />
**** Could be a bug in the old vistrail, but not sure<br />
* [DK] Subworkflows and uuids<br />
<br />
=== August 3, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Wrapper patches now stored as [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/vtk_patch.py python code]<br />
** Connection variables now lowercase<br />
** Can we use code as output variables?<br />
*** E.g. "vtk_dataset_mapper.GetOutput(0)"<br />
*** Could mutate output when output method is only supposed to be called once<br />
*** Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)<br />
<br />
=== July 27, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] DB bundle serialization implemented<br />
<br />
'''Items to Discuss'''<br />
* [TE] Finished using patch in script [https://github.com/VisTrails/VisTrails/tree/python-wrapper-script-patches python-wrapper-script-patches]<br />
** locale?<br />
** [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d new patching example] vs [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 old patching example]<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/parse.py#L135-L273 Example of new patches] <br />
** Legacy '.vtkInstance' is now added as a patch "self.vtkInstance=self" (see new patching example)<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/api.py New api for executing standalone modules]<br />
*** Accessing module was disabled for core/api, but a different api for this is probably better anyway.<br />
*** api.Package now gives [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d#file-new_patching-py-L26 directly callable module compute functions]<br />
*** Only works on modules not using interpreter functionality ('.interpreter', etc.)<br />
**** for zip file modules and others that need temp files and use the interpreter for this, need to modify the module, but could work around with python libraries for managing temp files<br />
** Fixed more renaming issues - [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/scripts.py#L63-L64 function argument names should not be replaced]<br />
** Still need to do something nicer for the patch formatting<br />
*** patch keys currently contain non-python-variable characters but this can be fixed<br />
*** Need to replace string template with python variables that is replacable using RedBaron<br />
**** Patches could then look like:<br />
<br />
def vtkInstance():<br />
self.vtkInstance = self<br />
<br />
=== July 20, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] New patch method for python-wrapper [https://gist.github.com/rexissimus/009b303e4cadcd4b8f2fec6cf848cb2b example]<br />
** A patch is a string template with self/input/output variables for patching a class method <br />
** Patches are stored in xml specification<br />
** Patches are assigned to methods in ClassSpec's<br />
** Used both for execution in vistrails and when exporting as script<br />
*** Executed with "exec" when run in vistrails<br />
*** Turns patches into natural calls in script<br />
** Wrapping VTK library no longer needed<br />
*** Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)<br />
** Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)<br />
** Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)<br />
** TODO's<br />
*** Add to export_as_script<br />
*** Patch Modules that represent Python functions as well<br />
<br />
* [vistrails-users] Read data from SQL example?<br />
<br />
=== July 13, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] bundles on sql-alchemy branch<br />
** Bundle, BundleMapping, and BundleSerializer<br />
** New BaseSerializers to deal with versions of directory, zip, db serializers<br />
** Can register the BundleSerializers with the base serializers<br />
** Translation between bundle versions<br />
** Bundle.add_object "just works" if an appropriate mapping was registered<br />
<br />
'''Items to Discuss'''<br />
* [TE] Simplified running module using API for scripting [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 example]<br />
** API Module can be executed using '''compute''' method<br />
*** Takes list of outputs to compute + inputs as keyword arguments<br />
* Status of general cell synchronization (vistrails-users)<br />
<br />
=== July 6, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] Working on bundles<br />
<br />
'''Items to Discuss'''<br />
* [TE] Using vistrails API in scripting [https://gist.github.com/rexissimus/034d9c4dedc3299a079e84981dedd8c7 example 1 (API with VTK)] [https://gist.github.com/rexissimus/a4a9ca6c08d4058c64967d2b35415fd5 example 2 (DownloadFile only)]<br />
** Call module in script using vistrails API<br />
** Can be used when no export method is defined.<br />
* [TE] [https://github.com/VisTrails/VisTrails/commit/c76c109349769fee1280f85c22678380a2f9cd39 Server news bug] Critical?<br />
**[RR] No, unreleased<br />
* [DK] New bundle organization<br />
** BundleMapping contains individual mappings from raw objects to BundleObj<br />
** Must be separate in order to have in-memory, separate from serializer settings<br />
<br />
=== June 29, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as python script<br />
** Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) [https://gist.github.com/rexissimus/16caf5200802e150296f62a0c7844c5a example]<br />
** Preludes (imports) still need improvements<br />
** Works with vtkInteractionHandler (clipping plane)<br />
** Resulting clipping plane is incorrect? (supposed to hide one side of the plane)<br />
** vtkRendererOutput Uses VTKRenderWindow as output<br />
*** Should support different output modes?<br />
*** Does not work with multiple outputs (Next cell is displayed when current is closed)<br />
*** Uses wrapped vtkInstance (as vvtk)<br />
**** Create new patch system with code that can be used both in vistrails and in a python script?<br />
<br />
=== June 22, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fix for upgraded analogies (PR [https://github.com/VisTrails/VisTrails/pull/1177 #1177])<br />
** Uses module location to find module remap (The best we can do right now?)<br />
** Also fixes remapped port names for connections to existing modules <br />
** May still fail on complex upgrades like adding a function on an upgraded module that has a different port name<br />
<br />
* [TE] Working on exporting wrapped libraries (VTK)<br />
<br />
=== June 15, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Added depth logic to python script exporter ([https://github.com/VisTrails/VisTrails/commit/50f220e4f6abe5a7bb8689f73f456f43df2dbc12 full commit])<br />
** Fully implements connection merging and list looping<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Making analogy work on upgraded workflows ([https://github.com/VisTrails/VisTrails/issues/1175 #1175])<br />
** Analogy only works We need to use the non-upgraded action chain<br />
** But pipelines need to be valid (matcher uses module portspecs)<br />
** Tried to get upgrade remap from upgrade action, but this seems impossible<br />
** Use module location instead?<br />
*** Location should be identical for one-to-one module upgrades<br />
<br />
=== June 1, 2016 ===<br />
<br />
'''Updates'''<br />
* VisTrails 2.2.4 announced (finally)<br />
** 5 survey replies<br />
** 47 downloads last week<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as script (python-wrapper)<br />
** to_python_script<br />
*** needs the pipeline (or at least the connections)<br />
**** Should not generate code for any possible VTK method (exporter detects which ports is used in the source)<br />
**** VTK needs to know connection order<br />
**** Adding functions directly to the script results in nicer code<br />
** Add support for multiple input connections<br />
** Add support for list looping<br />
<br />
=== May 25, 2016 ===<br />
<br />
'''Updates'''<br />
* Mailing-list fixed<br />
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])<br />
<br />
'''Items to Discuss'''<br />
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)<br />
* [TE] Union ports<br />
** Show union ports in pipeline (DEMO)<br />
<br />
=== May 18, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] mailing list still down - Julio working on it<br />
* [TE] Improved union port<br />
** New schema 1.0.5 adds portitem.union string attribute<br />
** Works with pipeline view, parameter exploration view, and mashup view<br />
** Should work well for matplotlib<br />
** Use with VTK ports?<br />
*** It uses [item_1,item_2] style ports usually with different tuple length.<br />
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]<br />
<br />
=== May 11, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?<br />
*** Shows single port for all ports with same sort_key<br />
*** Select specific type when adding/showing<br />
*** Only one type should be visible on the module at any time<br />
*** Making tuple parameters union types is probably too complicated<br />
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly) <br />
<br />
* [TE] Feature suggestion: Let connections connect to functions/parameters<br />
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters<br />
** Use separate views for ports and functions (like before)<br />
*** Allow rearranging of functions<br />
** Pros:<br />
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.<br />
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.<br />
**** Makes List module simpler<br />
*** Connect to parameter in tuple<br />
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds. <br />
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.<br />
** Cons:<br />
*** Interface more complicated and crowded<br />
*** More difficult for users to grasp?<br />
*** No immediate use case<br />
*** Makes no union type for tuple parameters a bigger problem.<br />
<br />
* [TE] Publish user survey?<br />
** No response from vistrails release yet<br />
** Not announced on vistrails-users yet.<br />
<br />
=== May 4, 2016 ===<br />
'''Updates'''<br />
* VisTrails 2.2.4 released<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
<br />
=== April 27, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Wrapping Bokeh<br />
** New method type "operation"<br />
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output<br />
**** Glyph sets function name and arguments as output<br />
***** Not serializable, but should always be computed together with the subject of the operation.<br />
**** Figure applies Glyph operation on itself<br />
*** Useful for matplotlib?<br />
**** matplotlib currently find actors by searching upstream?<br />
**** But this is more difficult since matplotlib is not function-based.<br />
** New "union" port type<br />
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value<br />
**** Uses only first value in tuple<br />
**** Bokeh have many properties that support (list, string, Float)<br />
*** Does not work with depth=1 types (e.g. Float of depth 1)<br />
**** Turn Depth 1 types with known type into lists?<br />
<br />
* [TE] New bugfix release<br />
** Finalize survey<br />
<br />
=== April 20, 2016 ===<br />
'''Updates'''<br />
* Merged upgrade fixes<br />
** Now has infinite loop test<br />
** loop test now correctly loops 5 times<br />
*** remap_module used wrong (current) package version when creating ModuleDescriptor<br />
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])<br />
<br />
'''Items to Discuss'''<br />
* New bugfix release<br />
** Merge usagestats<br />
** Need to finalize survey<br />
<br />
=== April 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?<br />
<br />
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])<br />
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]<br />
** Cherry-pick test to preserve source?<br />
** Need maxPipelineFixAttempts option?<br />
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.<br />
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?<br />
*** Would need to be moved into handle_invalid_pipeline<br />
<br />
=== April 6, 2016 ===<br />
'''Updates'''<br />
* [TE] Query View improvements<br />
** Done except for displaying "Version Info" and #1164<br />
* [RR] Working on finishing new interpreter (finally!)<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Try latest upgrade first, then previous.<br />
** If all fail, select latest upgrade?<br />
<br />
=== March 30, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Query View improvements<br />
** Unified do_version_switch and get_upgrades as validate_version()<br />
** Problems with query view using its own controller<br />
*** Fixed module info not being displayed for query results<br />
*** Still no version info for query results<br />
*** Still an issue with version query view not being updated<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== Older meetings ==<br />
* [[Development/2015]]<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=12578Development2016-10-19T14:52:49Z<p>Remi: /* October 19, 2016 */ Fix markup</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2016 ==<br />
=== October 19, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
New version 2.2-2 of Debian package, still [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=796994 wrong copyright], and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=748692 broken bundle installation]<br />
<br />
'''Items to Discuss'''<br />
<br />
=== October 5, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
<br />
* working toward a 3.0 release; [https://github.com/VisTrails/VisTrails/projects/1 project board]<br />
** unique ids<br />
** bundles<br />
** GUI fixes<br />
** wrapping updates<br />
** python3<br />
** interpreter?<br />
<br />
* command-line input parameters to provenance<br />
** [DK] we don't create a persisted workflow from those input params, i think<br />
** [DK] these are already defined as aliases so we treat them differently form a normal input param<br />
** [DK] still have the potential problem of allowing a user to view the workflow as it was run with a given set of aliases<br />
** Do the vistrails variables values get stored in provenance? (what if values change?)<br />
<br />
=== September 14, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] [http://aosabook.org/en/vistrails.html VisTrails] in [http://aosabook.org/ The Architecture of Open Source Applications] Vol I (2012)<br />
** Did we know/contribute to this?<br />
*** [DK] Yes. We wrote the text for the VisTrails chapter<br />
** Drafted response<br />
*** [DK] Looks good, will forward.<br />
<br />
=== August 31, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Went over vgc.poly.edu/vistrails.org config:<br />
** Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)<br />
** Fixed usersguide scripts (won't disappear again)<br />
** Moved some websites over to HTTPS<br />
** [https://docs.google.com/document/d/18YZewjR2YY9cQ9pzdA-nU9jbu2L7jYcPlDi3iaPadwA/edit Websites]<br />
* [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs ([http://vistrails.readthedocs.io/ currently there]) and users' guide under the same RTD "project"<br />
* [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that<br />
<br />
'''Items to Discuss'''<br />
* [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/<br />
** Get rid of more services? (svn, trac, ...)<br />
* [RR] Notebook execution works. How do we support more languages (than Python)?<br />
** Need: detection, metadata reader, code to input values<br />
* [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?<br />
<br />
=== August 17, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Fixed [https://github.com/VisTrails/VisTrails/commit/2cd42d156243067a7663bb0bf0156e5d87188c2f parameter order upgrade bug] that breaks VTK examples.<br />
* [TE] Matplotlib and sklearn are exportable using module-to-script api<br />
* [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.<br />
<br />
'''Items to Discuss'''<br />
* Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!<br />
* Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)<br />
* [TE] Wrap-up notes<br />
<br />
=== August 10, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Python Wrapper and scripting<br />
** All changes merged into python-wrapper branch<br />
*** Tests pass<br />
*** Still need to update sub-branches<br />
** Minor fixes<br />
*** vtkPlot3DReader upgrade bug<br />
*** Added tuple exporting<br />
** Most of vtk examples can be exported [https://gist.github.com/rexissimus/5ac7374cde8525d84f9443f019d234e2 example]<br />
*** Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary<br />
**** Could be a bug in the old vistrail, but not sure<br />
* [DK] Subworkflows and uuids<br />
<br />
=== August 3, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Wrapper patches now stored as [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/vtk_patch.py python code]<br />
** Connection variables now lowercase<br />
** Can we use code as output variables?<br />
*** E.g. "vtk_dataset_mapper.GetOutput(0)"<br />
*** Could mutate output when output method is only supposed to be called once<br />
*** Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)<br />
<br />
=== July 27, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] DB bundle serialization implemented<br />
<br />
'''Items to Discuss'''<br />
* [TE] Finished using patch in script [https://github.com/VisTrails/VisTrails/tree/python-wrapper-script-patches python-wrapper-script-patches]<br />
** locale?<br />
** [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d new patching example] vs [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 old patching example]<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/parse.py#L135-L273 Example of new patches] <br />
** Legacy '.vtkInstance' is now added as a patch "self.vtkInstance=self" (see new patching example)<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/api.py New api for executing standalone modules]<br />
*** Accessing module was disabled for core/api, but a different api for this is probably better anyway.<br />
*** api.Package now gives [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d#file-new_patching-py-L26 directly callable module compute functions]<br />
*** Only works on modules not using interpreter functionality ('.interpreter', etc.)<br />
**** for zip file modules and others that need temp files and use the interpreter for this, need to modify the module, but could work around with python libraries for managing temp files<br />
** Fixed more renaming issues - [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/scripts.py#L63-L64 function argument names should not be replaced]<br />
** Still need to do something nicer for the patch formatting<br />
*** patch keys currently contain non-python-variable characters but this can be fixed<br />
*** Need to replace string template with python variables that is replacable using RedBaron<br />
**** Patches could then look like:<br />
<br />
def vtkInstance():<br />
self.vtkInstance = self<br />
<br />
=== July 20, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] New patch method for python-wrapper [https://gist.github.com/rexissimus/009b303e4cadcd4b8f2fec6cf848cb2b example]<br />
** A patch is a string template with self/input/output variables for patching a class method <br />
** Patches are stored in xml specification<br />
** Patches are assigned to methods in ClassSpec's<br />
** Used both for execution in vistrails and when exporting as script<br />
*** Executed with "exec" when run in vistrails<br />
*** Turns patches into natural calls in script<br />
** Wrapping VTK library no longer needed<br />
*** Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)<br />
** Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)<br />
** Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)<br />
** TODO's<br />
*** Add to export_as_script<br />
*** Patch Modules that represent Python functions as well<br />
<br />
* [vistrails-users] Read data from SQL example?<br />
<br />
=== July 13, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] bundles on sql-alchemy branch<br />
** Bundle, BundleMapping, and BundleSerializer<br />
** New BaseSerializers to deal with versions of directory, zip, db serializers<br />
** Can register the BundleSerializers with the base serializers<br />
** Translation between bundle versions<br />
** Bundle.add_object "just works" if an appropriate mapping was registered<br />
<br />
'''Items to Discuss'''<br />
* [TE] Simplified running module using API for scripting [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 example]<br />
** API Module can be executed using '''compute''' method<br />
*** Takes list of outputs to compute + inputs as keyword arguments<br />
* Status of general cell synchronization (vistrails-users)<br />
<br />
=== July 6, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] Working on bundles<br />
<br />
'''Items to Discuss'''<br />
* [TE] Using vistrails API in scripting [https://gist.github.com/rexissimus/034d9c4dedc3299a079e84981dedd8c7 example 1 (API with VTK)] [https://gist.github.com/rexissimus/a4a9ca6c08d4058c64967d2b35415fd5 example 2 (DownloadFile only)]<br />
** Call module in script using vistrails API<br />
** Can be used when no export method is defined.<br />
* [TE] [https://github.com/VisTrails/VisTrails/commit/c76c109349769fee1280f85c22678380a2f9cd39 Server news bug] Critical?<br />
**[RR] No, unreleased<br />
* [DK] New bundle organization<br />
** BundleMapping contains individual mappings from raw objects to BundleObj<br />
** Must be separate in order to have in-memory, separate from serializer settings<br />
<br />
=== June 29, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as python script<br />
** Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) [https://gist.github.com/rexissimus/16caf5200802e150296f62a0c7844c5a example]<br />
** Preludes (imports) still need improvements<br />
** Works with vtkInteractionHandler (clipping plane)<br />
** Resulting clipping plane is incorrect? (supposed to hide one side of the plane)<br />
** vtkRendererOutput Uses VTKRenderWindow as output<br />
*** Should support different output modes?<br />
*** Does not work with multiple outputs (Next cell is displayed when current is closed)<br />
*** Uses wrapped vtkInstance (as vvtk)<br />
**** Create new patch system with code that can be used both in vistrails and in a python script?<br />
<br />
=== June 22, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fix for upgraded analogies (PR [https://github.com/VisTrails/VisTrails/pull/1177 #1177])<br />
** Uses module location to find module remap (The best we can do right now?)<br />
** Also fixes remapped port names for connections to existing modules <br />
** May still fail on complex upgrades like adding a function on an upgraded module that has a different port name<br />
<br />
* [TE] Working on exporting wrapped libraries (VTK)<br />
<br />
=== June 15, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Added depth logic to python script exporter ([https://github.com/VisTrails/VisTrails/commit/50f220e4f6abe5a7bb8689f73f456f43df2dbc12 full commit])<br />
** Fully implements connection merging and list looping<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Making analogy work on upgraded workflows ([https://github.com/VisTrails/VisTrails/issues/1175 #1175])<br />
** Analogy only works We need to use the non-upgraded action chain<br />
** But pipelines need to be valid (matcher uses module portspecs)<br />
** Tried to get upgrade remap from upgrade action, but this seems impossible<br />
** Use module location instead?<br />
*** Location should be identical for one-to-one module upgrades<br />
<br />
=== June 1, 2016 ===<br />
<br />
'''Updates'''<br />
* VisTrails 2.2.4 announced (finally)<br />
** 5 survey replies<br />
** 47 downloads last week<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as script (python-wrapper)<br />
** to_python_script<br />
*** needs the pipeline (or at least the connections)<br />
**** Should not generate code for any possible VTK method (exporter detects which ports is used in the source)<br />
**** VTK needs to know connection order<br />
**** Adding functions directly to the script results in nicer code<br />
** Add support for multiple input connections<br />
** Add support for list looping<br />
<br />
=== May 25, 2016 ===<br />
<br />
'''Updates'''<br />
* Mailing-list fixed<br />
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])<br />
<br />
'''Items to Discuss'''<br />
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)<br />
* [TE] Union ports<br />
** Show union ports in pipeline (DEMO)<br />
<br />
=== May 18, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] mailing list still down - Julio working on it<br />
* [TE] Improved union port<br />
** New schema 1.0.5 adds portitem.union string attribute<br />
** Works with pipeline view, parameter exploration view, and mashup view<br />
** Should work well for matplotlib<br />
** Use with VTK ports?<br />
*** It uses [item_1,item_2] style ports usually with different tuple length.<br />
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]<br />
<br />
=== May 11, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?<br />
*** Shows single port for all ports with same sort_key<br />
*** Select specific type when adding/showing<br />
*** Only one type should be visible on the module at any time<br />
*** Making tuple parameters union types is probably too complicated<br />
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly) <br />
<br />
* [TE] Feature suggestion: Let connections connect to functions/parameters<br />
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters<br />
** Use separate views for ports and functions (like before)<br />
*** Allow rearranging of functions<br />
** Pros:<br />
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.<br />
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.<br />
**** Makes List module simpler<br />
*** Connect to parameter in tuple<br />
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds. <br />
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.<br />
** Cons:<br />
*** Interface more complicated and crowded<br />
*** More difficult for users to grasp?<br />
*** No immediate use case<br />
*** Makes no union type for tuple parameters a bigger problem.<br />
<br />
* [TE] Publish user survey?<br />
** No response from vistrails release yet<br />
** Not announced on vistrails-users yet.<br />
<br />
=== May 4, 2016 ===<br />
'''Updates'''<br />
* VisTrails 2.2.4 released<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
<br />
=== April 27, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Wrapping Bokeh<br />
** New method type "operation"<br />
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output<br />
**** Glyph sets function name and arguments as output<br />
***** Not serializable, but should always be computed together with the subject of the operation.<br />
**** Figure applies Glyph operation on itself<br />
*** Useful for matplotlib?<br />
**** matplotlib currently find actors by searching upstream?<br />
**** But this is more difficult since matplotlib is not function-based.<br />
** New "union" port type<br />
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value<br />
**** Uses only first value in tuple<br />
**** Bokeh have many properties that support (list, string, Float)<br />
*** Does not work with depth=1 types (e.g. Float of depth 1)<br />
**** Turn Depth 1 types with known type into lists?<br />
<br />
* [TE] New bugfix release<br />
** Finalize survey<br />
<br />
=== April 20, 2016 ===<br />
'''Updates'''<br />
* Merged upgrade fixes<br />
** Now has infinite loop test<br />
** loop test now correctly loops 5 times<br />
*** remap_module used wrong (current) package version when creating ModuleDescriptor<br />
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])<br />
<br />
'''Items to Discuss'''<br />
* New bugfix release<br />
** Merge usagestats<br />
** Need to finalize survey<br />
<br />
=== April 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?<br />
<br />
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])<br />
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]<br />
** Cherry-pick test to preserve source?<br />
** Need maxPipelineFixAttempts option?<br />
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.<br />
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?<br />
*** Would need to be moved into handle_invalid_pipeline<br />
<br />
=== April 6, 2016 ===<br />
'''Updates'''<br />
* [TE] Query View improvements<br />
** Done except for displaying "Version Info" and #1164<br />
* [RR] Working on finishing new interpreter (finally!)<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Try latest upgrade first, then previous.<br />
** If all fail, select latest upgrade?<br />
<br />
=== March 30, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Query View improvements<br />
** Unified do_version_switch and get_upgrades as validate_version()<br />
** Problems with query view using its own controller<br />
*** Fixed module info not being displayed for query results<br />
*** Still no version info for query results<br />
*** Still an issue with version query view not being updated<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== Older meetings ==<br />
* [[Development/2015]]<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=12576Development2016-10-18T20:18:12Z<p>Remi: /* October 19, 2016 */</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2016 ==<br />
=== October 19, 2016 ===<br />
<br />
'''Updates'''<br />
New version 2.2-2 of Debian package, still [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=796994 wrong copyright], and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=748692 broken bundle installation]<br />
<br />
'''Items to Discuss'''<br />
<br />
=== October 5, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
<br />
* working toward a 3.0 release; [https://github.com/VisTrails/VisTrails/projects/1 project board]<br />
** unique ids<br />
** bundles<br />
** GUI fixes<br />
** wrapping updates<br />
** python3<br />
** interpreter?<br />
<br />
* command-line input parameters to provenance<br />
** [DK] we don't create a persisted workflow from those input params, i think<br />
** [DK] these are already defined as aliases so we treat them differently form a normal input param<br />
** [DK] still have the potential problem of allowing a user to view the workflow as it was run with a given set of aliases<br />
** Do the vistrails variables values get stored in provenance? (what if values change?)<br />
<br />
=== September 14, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] [http://aosabook.org/en/vistrails.html VisTrails] in [http://aosabook.org/ The Architecture of Open Source Applications] Vol I (2012)<br />
** Did we know/contribute to this?<br />
*** [DK] Yes. We wrote the text for the VisTrails chapter<br />
** Drafted response<br />
*** [DK] Looks good, will forward.<br />
<br />
=== August 31, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Went over vgc.poly.edu/vistrails.org config:<br />
** Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)<br />
** Fixed usersguide scripts (won't disappear again)<br />
** Moved some websites over to HTTPS<br />
** [https://docs.google.com/document/d/18YZewjR2YY9cQ9pzdA-nU9jbu2L7jYcPlDi3iaPadwA/edit Websites]<br />
* [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs ([http://vistrails.readthedocs.io/ currently there]) and users' guide under the same RTD "project"<br />
* [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that<br />
<br />
'''Items to Discuss'''<br />
* [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/<br />
** Get rid of more services? (svn, trac, ...)<br />
* [RR] Notebook execution works. How do we support more languages (than Python)?<br />
** Need: detection, metadata reader, code to input values<br />
* [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?<br />
<br />
=== August 17, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Fixed [https://github.com/VisTrails/VisTrails/commit/2cd42d156243067a7663bb0bf0156e5d87188c2f parameter order upgrade bug] that breaks VTK examples.<br />
* [TE] Matplotlib and sklearn are exportable using module-to-script api<br />
* [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.<br />
<br />
'''Items to Discuss'''<br />
* Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!<br />
* Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)<br />
* [TE] Wrap-up notes<br />
<br />
=== August 10, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Python Wrapper and scripting<br />
** All changes merged into python-wrapper branch<br />
*** Tests pass<br />
*** Still need to update sub-branches<br />
** Minor fixes<br />
*** vtkPlot3DReader upgrade bug<br />
*** Added tuple exporting<br />
** Most of vtk examples can be exported [https://gist.github.com/rexissimus/5ac7374cde8525d84f9443f019d234e2 example]<br />
*** Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary<br />
**** Could be a bug in the old vistrail, but not sure<br />
* [DK] Subworkflows and uuids<br />
<br />
=== August 3, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Wrapper patches now stored as [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/vtk_patch.py python code]<br />
** Connection variables now lowercase<br />
** Can we use code as output variables?<br />
*** E.g. "vtk_dataset_mapper.GetOutput(0)"<br />
*** Could mutate output when output method is only supposed to be called once<br />
*** Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)<br />
<br />
=== July 27, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] DB bundle serialization implemented<br />
<br />
'''Items to Discuss'''<br />
* [TE] Finished using patch in script [https://github.com/VisTrails/VisTrails/tree/python-wrapper-script-patches python-wrapper-script-patches]<br />
** locale?<br />
** [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d new patching example] vs [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 old patching example]<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/parse.py#L135-L273 Example of new patches] <br />
** Legacy '.vtkInstance' is now added as a patch "self.vtkInstance=self" (see new patching example)<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/api.py New api for executing standalone modules]<br />
*** Accessing module was disabled for core/api, but a different api for this is probably better anyway.<br />
*** api.Package now gives [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d#file-new_patching-py-L26 directly callable module compute functions]<br />
*** Only works on modules not using interpreter functionality ('.interpreter', etc.)<br />
**** for zip file modules and others that need temp files and use the interpreter for this, need to modify the module, but could work around with python libraries for managing temp files<br />
** Fixed more renaming issues - [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/scripts.py#L63-L64 function argument names should not be replaced]<br />
** Still need to do something nicer for the patch formatting<br />
*** patch keys currently contain non-python-variable characters but this can be fixed<br />
*** Need to replace string template with python variables that is replacable using RedBaron<br />
**** Patches could then look like:<br />
<br />
def vtkInstance():<br />
self.vtkInstance = self<br />
<br />
=== July 20, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] New patch method for python-wrapper [https://gist.github.com/rexissimus/009b303e4cadcd4b8f2fec6cf848cb2b example]<br />
** A patch is a string template with self/input/output variables for patching a class method <br />
** Patches are stored in xml specification<br />
** Patches are assigned to methods in ClassSpec's<br />
** Used both for execution in vistrails and when exporting as script<br />
*** Executed with "exec" when run in vistrails<br />
*** Turns patches into natural calls in script<br />
** Wrapping VTK library no longer needed<br />
*** Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)<br />
** Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)<br />
** Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)<br />
** TODO's<br />
*** Add to export_as_script<br />
*** Patch Modules that represent Python functions as well<br />
<br />
* [vistrails-users] Read data from SQL example?<br />
<br />
=== July 13, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] bundles on sql-alchemy branch<br />
** Bundle, BundleMapping, and BundleSerializer<br />
** New BaseSerializers to deal with versions of directory, zip, db serializers<br />
** Can register the BundleSerializers with the base serializers<br />
** Translation between bundle versions<br />
** Bundle.add_object "just works" if an appropriate mapping was registered<br />
<br />
'''Items to Discuss'''<br />
* [TE] Simplified running module using API for scripting [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 example]<br />
** API Module can be executed using '''compute''' method<br />
*** Takes list of outputs to compute + inputs as keyword arguments<br />
* Status of general cell synchronization (vistrails-users)<br />
<br />
=== July 6, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] Working on bundles<br />
<br />
'''Items to Discuss'''<br />
* [TE] Using vistrails API in scripting [https://gist.github.com/rexissimus/034d9c4dedc3299a079e84981dedd8c7 example 1 (API with VTK)] [https://gist.github.com/rexissimus/a4a9ca6c08d4058c64967d2b35415fd5 example 2 (DownloadFile only)]<br />
** Call module in script using vistrails API<br />
** Can be used when no export method is defined.<br />
* [TE] [https://github.com/VisTrails/VisTrails/commit/c76c109349769fee1280f85c22678380a2f9cd39 Server news bug] Critical?<br />
**[RR] No, unreleased<br />
* [DK] New bundle organization<br />
** BundleMapping contains individual mappings from raw objects to BundleObj<br />
** Must be separate in order to have in-memory, separate from serializer settings<br />
<br />
=== June 29, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as python script<br />
** Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) [https://gist.github.com/rexissimus/16caf5200802e150296f62a0c7844c5a example]<br />
** Preludes (imports) still need improvements<br />
** Works with vtkInteractionHandler (clipping plane)<br />
** Resulting clipping plane is incorrect? (supposed to hide one side of the plane)<br />
** vtkRendererOutput Uses VTKRenderWindow as output<br />
*** Should support different output modes?<br />
*** Does not work with multiple outputs (Next cell is displayed when current is closed)<br />
*** Uses wrapped vtkInstance (as vvtk)<br />
**** Create new patch system with code that can be used both in vistrails and in a python script?<br />
<br />
=== June 22, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fix for upgraded analogies (PR [https://github.com/VisTrails/VisTrails/pull/1177 #1177])<br />
** Uses module location to find module remap (The best we can do right now?)<br />
** Also fixes remapped port names for connections to existing modules <br />
** May still fail on complex upgrades like adding a function on an upgraded module that has a different port name<br />
<br />
* [TE] Working on exporting wrapped libraries (VTK)<br />
<br />
=== June 15, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Added depth logic to python script exporter ([https://github.com/VisTrails/VisTrails/commit/50f220e4f6abe5a7bb8689f73f456f43df2dbc12 full commit])<br />
** Fully implements connection merging and list looping<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Making analogy work on upgraded workflows ([https://github.com/VisTrails/VisTrails/issues/1175 #1175])<br />
** Analogy only works We need to use the non-upgraded action chain<br />
** But pipelines need to be valid (matcher uses module portspecs)<br />
** Tried to get upgrade remap from upgrade action, but this seems impossible<br />
** Use module location instead?<br />
*** Location should be identical for one-to-one module upgrades<br />
<br />
=== June 1, 2016 ===<br />
<br />
'''Updates'''<br />
* VisTrails 2.2.4 announced (finally)<br />
** 5 survey replies<br />
** 47 downloads last week<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as script (python-wrapper)<br />
** to_python_script<br />
*** needs the pipeline (or at least the connections)<br />
**** Should not generate code for any possible VTK method (exporter detects which ports is used in the source)<br />
**** VTK needs to know connection order<br />
**** Adding functions directly to the script results in nicer code<br />
** Add support for multiple input connections<br />
** Add support for list looping<br />
<br />
=== May 25, 2016 ===<br />
<br />
'''Updates'''<br />
* Mailing-list fixed<br />
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])<br />
<br />
'''Items to Discuss'''<br />
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)<br />
* [TE] Union ports<br />
** Show union ports in pipeline (DEMO)<br />
<br />
=== May 18, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] mailing list still down - Julio working on it<br />
* [TE] Improved union port<br />
** New schema 1.0.5 adds portitem.union string attribute<br />
** Works with pipeline view, parameter exploration view, and mashup view<br />
** Should work well for matplotlib<br />
** Use with VTK ports?<br />
*** It uses [item_1,item_2] style ports usually with different tuple length.<br />
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]<br />
<br />
=== May 11, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?<br />
*** Shows single port for all ports with same sort_key<br />
*** Select specific type when adding/showing<br />
*** Only one type should be visible on the module at any time<br />
*** Making tuple parameters union types is probably too complicated<br />
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly) <br />
<br />
* [TE] Feature suggestion: Let connections connect to functions/parameters<br />
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters<br />
** Use separate views for ports and functions (like before)<br />
*** Allow rearranging of functions<br />
** Pros:<br />
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.<br />
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.<br />
**** Makes List module simpler<br />
*** Connect to parameter in tuple<br />
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds. <br />
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.<br />
** Cons:<br />
*** Interface more complicated and crowded<br />
*** More difficult for users to grasp?<br />
*** No immediate use case<br />
*** Makes no union type for tuple parameters a bigger problem.<br />
<br />
* [TE] Publish user survey?<br />
** No response from vistrails release yet<br />
** Not announced on vistrails-users yet.<br />
<br />
=== May 4, 2016 ===<br />
'''Updates'''<br />
* VisTrails 2.2.4 released<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
<br />
=== April 27, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Wrapping Bokeh<br />
** New method type "operation"<br />
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output<br />
**** Glyph sets function name and arguments as output<br />
***** Not serializable, but should always be computed together with the subject of the operation.<br />
**** Figure applies Glyph operation on itself<br />
*** Useful for matplotlib?<br />
**** matplotlib currently find actors by searching upstream?<br />
**** But this is more difficult since matplotlib is not function-based.<br />
** New "union" port type<br />
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value<br />
**** Uses only first value in tuple<br />
**** Bokeh have many properties that support (list, string, Float)<br />
*** Does not work with depth=1 types (e.g. Float of depth 1)<br />
**** Turn Depth 1 types with known type into lists?<br />
<br />
* [TE] New bugfix release<br />
** Finalize survey<br />
<br />
=== April 20, 2016 ===<br />
'''Updates'''<br />
* Merged upgrade fixes<br />
** Now has infinite loop test<br />
** loop test now correctly loops 5 times<br />
*** remap_module used wrong (current) package version when creating ModuleDescriptor<br />
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])<br />
<br />
'''Items to Discuss'''<br />
* New bugfix release<br />
** Merge usagestats<br />
** Need to finalize survey<br />
<br />
=== April 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?<br />
<br />
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])<br />
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]<br />
** Cherry-pick test to preserve source?<br />
** Need maxPipelineFixAttempts option?<br />
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.<br />
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?<br />
*** Would need to be moved into handle_invalid_pipeline<br />
<br />
=== April 6, 2016 ===<br />
'''Updates'''<br />
* [TE] Query View improvements<br />
** Done except for displaying "Version Info" and #1164<br />
* [RR] Working on finishing new interpreter (finally!)<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Try latest upgrade first, then previous.<br />
** If all fail, select latest upgrade?<br />
<br />
=== March 30, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Query View improvements<br />
** Unified do_version_switch and get_upgrades as validate_version()<br />
** Problems with query view using its own controller<br />
*** Fixed module info not being displayed for query results<br />
*** Still no version info for query results<br />
*** Still an issue with version query view not being updated<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== Older meetings ==<br />
* [[Development/2015]]<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=12539Development2016-10-05T16:05:14Z<p>Remi: /* October 5, 2016 */</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2016 ==<br />
=== October 5, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
<br />
* working toward a 3.0 release; [https://github.com/VisTrails/VisTrails/projects/1 project board]<br />
** unique ids<br />
** bundles<br />
** GUI fixes<br />
** wrapping updates<br />
** python3<br />
** interpreter?<br />
<br />
* command-line input parameters to provenance<br />
** [DK] we don't create a persisted workflow from those input params, i think<br />
** [DK] these are already defined as aliases so we treat them differently form a normal input param<br />
** [DK] still have the potential problem of allowing a user to view the workflow as it was run with a given set of aliases<br />
** Do the vistrails variables values get stored in provenance? (what if values change?)<br />
<br />
=== September 14, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] [http://aosabook.org/en/vistrails.html VisTrails] in [http://aosabook.org/ The Architecture of Open Source Applications] Vol I (2012)<br />
** Did we know/contribute to this?<br />
*** [DK] Yes. We wrote the text for the VisTrails chapter<br />
** Drafted response<br />
*** [DK] Looks good, will forward.<br />
<br />
=== August 31, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Went over vgc.poly.edu/vistrails.org config:<br />
** Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)<br />
** Fixed usersguide scripts (won't disappear again)<br />
** Moved some websites over to HTTPS<br />
** [https://docs.google.com/document/d/18YZewjR2YY9cQ9pzdA-nU9jbu2L7jYcPlDi3iaPadwA/edit Websites]<br />
* [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs ([http://vistrails.readthedocs.io/ currently there]) and users' guide under the same RTD "project"<br />
* [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that<br />
<br />
'''Items to Discuss'''<br />
* [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/<br />
** Get rid of more services? (svn, trac, ...)<br />
* [RR] Notebook execution works. How do we support more languages (than Python)?<br />
** Need: detection, metadata reader, code to input values<br />
* [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?<br />
<br />
=== August 17, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Fixed [https://github.com/VisTrails/VisTrails/commit/2cd42d156243067a7663bb0bf0156e5d87188c2f parameter order upgrade bug] that breaks VTK examples.<br />
* [TE] Matplotlib and sklearn are exportable using module-to-script api<br />
* [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.<br />
<br />
'''Items to Discuss'''<br />
* Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!<br />
* Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)<br />
* [TE] Wrap-up notes<br />
<br />
=== August 10, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Python Wrapper and scripting<br />
** All changes merged into python-wrapper branch<br />
*** Tests pass<br />
*** Still need to update sub-branches<br />
** Minor fixes<br />
*** vtkPlot3DReader upgrade bug<br />
*** Added tuple exporting<br />
** Most of vtk examples can be exported [https://gist.github.com/rexissimus/5ac7374cde8525d84f9443f019d234e2 example]<br />
*** Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary<br />
**** Could be a bug in the old vistrail, but not sure<br />
* [DK] Subworkflows and uuids<br />
<br />
=== August 3, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Wrapper patches now stored as [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/vtk_patch.py python code]<br />
** Connection variables now lowercase<br />
** Can we use code as output variables?<br />
*** E.g. "vtk_dataset_mapper.GetOutput(0)"<br />
*** Could mutate output when output method is only supposed to be called once<br />
*** Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)<br />
<br />
=== July 27, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] DB bundle serialization implemented<br />
<br />
'''Items to Discuss'''<br />
* [TE] Finished using patch in script [https://github.com/VisTrails/VisTrails/tree/python-wrapper-script-patches python-wrapper-script-patches]<br />
** locale?<br />
** [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d new patching example] vs [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 old patching example]<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/parse.py#L135-L273 Example of new patches] <br />
** Legacy '.vtkInstance' is now added as a patch "self.vtkInstance=self" (see new patching example)<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/api.py New api for executing standalone modules]<br />
*** Accessing module was disabled for core/api, but a different api for this is probably better anyway.<br />
*** api.Package now gives [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d#file-new_patching-py-L26 directly callable module compute functions]<br />
*** Only works on modules not using interpreter functionality ('.interpreter', etc.)<br />
**** for zip file modules and others that need temp files and use the interpreter for this, need to modify the module, but could work around with python libraries for managing temp files<br />
** Fixed more renaming issues - [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/scripts.py#L63-L64 function argument names should not be replaced]<br />
** Still need to do something nicer for the patch formatting<br />
*** patch keys currently contain non-python-variable characters but this can be fixed<br />
*** Need to replace string template with python variables that is replacable using RedBaron<br />
**** Patches could then look like:<br />
<br />
def vtkInstance():<br />
self.vtkInstance = self<br />
<br />
=== July 20, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] New patch method for python-wrapper [https://gist.github.com/rexissimus/009b303e4cadcd4b8f2fec6cf848cb2b example]<br />
** A patch is a string template with self/input/output variables for patching a class method <br />
** Patches are stored in xml specification<br />
** Patches are assigned to methods in ClassSpec's<br />
** Used both for execution in vistrails and when exporting as script<br />
*** Executed with "exec" when run in vistrails<br />
*** Turns patches into natural calls in script<br />
** Wrapping VTK library no longer needed<br />
*** Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)<br />
** Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)<br />
** Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)<br />
** TODO's<br />
*** Add to export_as_script<br />
*** Patch Modules that represent Python functions as well<br />
<br />
* [vistrails-users] Read data from SQL example?<br />
<br />
=== July 13, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] bundles on sql-alchemy branch<br />
** Bundle, BundleMapping, and BundleSerializer<br />
** New BaseSerializers to deal with versions of directory, zip, db serializers<br />
** Can register the BundleSerializers with the base serializers<br />
** Translation between bundle versions<br />
** Bundle.add_object "just works" if an appropriate mapping was registered<br />
<br />
'''Items to Discuss'''<br />
* [TE] Simplified running module using API for scripting [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 example]<br />
** API Module can be executed using '''compute''' method<br />
*** Takes list of outputs to compute + inputs as keyword arguments<br />
* Status of general cell synchronization (vistrails-users)<br />
<br />
=== July 6, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] Working on bundles<br />
<br />
'''Items to Discuss'''<br />
* [TE] Using vistrails API in scripting [https://gist.github.com/rexissimus/034d9c4dedc3299a079e84981dedd8c7 example 1 (API with VTK)] [https://gist.github.com/rexissimus/a4a9ca6c08d4058c64967d2b35415fd5 example 2 (DownloadFile only)]<br />
** Call module in script using vistrails API<br />
** Can be used when no export method is defined.<br />
* [TE] [https://github.com/VisTrails/VisTrails/commit/c76c109349769fee1280f85c22678380a2f9cd39 Server news bug] Critical?<br />
**[RR] No, unreleased<br />
* [DK] New bundle organization<br />
** BundleMapping contains individual mappings from raw objects to BundleObj<br />
** Must be separate in order to have in-memory, separate from serializer settings<br />
<br />
=== June 29, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as python script<br />
** Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) [https://gist.github.com/rexissimus/16caf5200802e150296f62a0c7844c5a example]<br />
** Preludes (imports) still need improvements<br />
** Works with vtkInteractionHandler (clipping plane)<br />
** Resulting clipping plane is incorrect? (supposed to hide one side of the plane)<br />
** vtkRendererOutput Uses VTKRenderWindow as output<br />
*** Should support different output modes?<br />
*** Does not work with multiple outputs (Next cell is displayed when current is closed)<br />
*** Uses wrapped vtkInstance (as vvtk)<br />
**** Create new patch system with code that can be used both in vistrails and in a python script?<br />
<br />
=== June 22, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fix for upgraded analogies (PR [https://github.com/VisTrails/VisTrails/pull/1177 #1177])<br />
** Uses module location to find module remap (The best we can do right now?)<br />
** Also fixes remapped port names for connections to existing modules <br />
** May still fail on complex upgrades like adding a function on an upgraded module that has a different port name<br />
<br />
* [TE] Working on exporting wrapped libraries (VTK)<br />
<br />
=== June 15, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Added depth logic to python script exporter ([https://github.com/VisTrails/VisTrails/commit/50f220e4f6abe5a7bb8689f73f456f43df2dbc12 full commit])<br />
** Fully implements connection merging and list looping<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Making analogy work on upgraded workflows ([https://github.com/VisTrails/VisTrails/issues/1175 #1175])<br />
** Analogy only works We need to use the non-upgraded action chain<br />
** But pipelines need to be valid (matcher uses module portspecs)<br />
** Tried to get upgrade remap from upgrade action, but this seems impossible<br />
** Use module location instead?<br />
*** Location should be identical for one-to-one module upgrades<br />
<br />
=== June 1, 2016 ===<br />
<br />
'''Updates'''<br />
* VisTrails 2.2.4 announced (finally)<br />
** 5 survey replies<br />
** 47 downloads last week<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as script (python-wrapper)<br />
** to_python_script<br />
*** needs the pipeline (or at least the connections)<br />
**** Should not generate code for any possible VTK method (exporter detects which ports is used in the source)<br />
**** VTK needs to know connection order<br />
**** Adding functions directly to the script results in nicer code<br />
** Add support for multiple input connections<br />
** Add support for list looping<br />
<br />
=== May 25, 2016 ===<br />
<br />
'''Updates'''<br />
* Mailing-list fixed<br />
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])<br />
<br />
'''Items to Discuss'''<br />
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)<br />
* [TE] Union ports<br />
** Show union ports in pipeline (DEMO)<br />
<br />
=== May 18, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] mailing list still down - Julio working on it<br />
* [TE] Improved union port<br />
** New schema 1.0.5 adds portitem.union string attribute<br />
** Works with pipeline view, parameter exploration view, and mashup view<br />
** Should work well for matplotlib<br />
** Use with VTK ports?<br />
*** It uses [item_1,item_2] style ports usually with different tuple length.<br />
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]<br />
<br />
=== May 11, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?<br />
*** Shows single port for all ports with same sort_key<br />
*** Select specific type when adding/showing<br />
*** Only one type should be visible on the module at any time<br />
*** Making tuple parameters union types is probably too complicated<br />
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly) <br />
<br />
* [TE] Feature suggestion: Let connections connect to functions/parameters<br />
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters<br />
** Use separate views for ports and functions (like before)<br />
*** Allow rearranging of functions<br />
** Pros:<br />
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.<br />
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.<br />
**** Makes List module simpler<br />
*** Connect to parameter in tuple<br />
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds. <br />
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.<br />
** Cons:<br />
*** Interface more complicated and crowded<br />
*** More difficult for users to grasp?<br />
*** No immediate use case<br />
*** Makes no union type for tuple parameters a bigger problem.<br />
<br />
* [TE] Publish user survey?<br />
** No response from vistrails release yet<br />
** Not announced on vistrails-users yet.<br />
<br />
=== May 4, 2016 ===<br />
'''Updates'''<br />
* VisTrails 2.2.4 released<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
<br />
=== April 27, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Wrapping Bokeh<br />
** New method type "operation"<br />
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output<br />
**** Glyph sets function name and arguments as output<br />
***** Not serializable, but should always be computed together with the subject of the operation.<br />
**** Figure applies Glyph operation on itself<br />
*** Useful for matplotlib?<br />
**** matplotlib currently find actors by searching upstream?<br />
**** But this is more difficult since matplotlib is not function-based.<br />
** New "union" port type<br />
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value<br />
**** Uses only first value in tuple<br />
**** Bokeh have many properties that support (list, string, Float)<br />
*** Does not work with depth=1 types (e.g. Float of depth 1)<br />
**** Turn Depth 1 types with known type into lists?<br />
<br />
* [TE] New bugfix release<br />
** Finalize survey<br />
<br />
=== April 20, 2016 ===<br />
'''Updates'''<br />
* Merged upgrade fixes<br />
** Now has infinite loop test<br />
** loop test now correctly loops 5 times<br />
*** remap_module used wrong (current) package version when creating ModuleDescriptor<br />
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])<br />
<br />
'''Items to Discuss'''<br />
* New bugfix release<br />
** Merge usagestats<br />
** Need to finalize survey<br />
<br />
=== April 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?<br />
<br />
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])<br />
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]<br />
** Cherry-pick test to preserve source?<br />
** Need maxPipelineFixAttempts option?<br />
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.<br />
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?<br />
*** Would need to be moved into handle_invalid_pipeline<br />
<br />
=== April 6, 2016 ===<br />
'''Updates'''<br />
* [TE] Query View improvements<br />
** Done except for displaying "Version Info" and #1164<br />
* [RR] Working on finishing new interpreter (finally!)<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Try latest upgrade first, then previous.<br />
** If all fail, select latest upgrade?<br />
<br />
=== March 30, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Query View improvements<br />
** Unified do_version_switch and get_upgrades as validate_version()<br />
** Problems with query view using its own controller<br />
*** Fixed module info not being displayed for query results<br />
*** Still no version info for query results<br />
*** Still an issue with version query view not being updated<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== Older meetings ==<br />
* [[Development/2015]]<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=12443Development2016-09-14T14:03:14Z<p>Remi: /* September 14, 2016 */</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2016 ==<br />
=== September 14, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] [http://aosabook.org/en/vistrails.html VisTrails] in [http://aosabook.org/ The Architecture of Open Source Applications] Vol I (2012)<br />
** Did we know/contribute to this?<br />
** Drafted response<br />
<br />
=== August 31, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Went over vgc.poly.edu/vistrails.org config:<br />
** Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)<br />
** Fixed usersguide scripts (won't disappear again)<br />
** Moved some websites over to HTTPS<br />
** [https://docs.google.com/document/d/18YZewjR2YY9cQ9pzdA-nU9jbu2L7jYcPlDi3iaPadwA/edit Websites]<br />
* [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs ([http://vistrails.readthedocs.io/ currently there]) and users' guide under the same RTD "project"<br />
* [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that<br />
<br />
'''Items to Discuss'''<br />
* [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/<br />
** Get rid of more services? (svn, trac, ...)<br />
* [RR] Notebook execution works. How do we support more languages (than Python)?<br />
** Need: detection, metadata reader, code to input values<br />
* [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?<br />
<br />
=== August 17, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Fixed [https://github.com/VisTrails/VisTrails/commit/2cd42d156243067a7663bb0bf0156e5d87188c2f parameter order upgrade bug] that breaks VTK examples.<br />
* [TE] Matplotlib and sklearn are exportable using module-to-script api<br />
* [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.<br />
<br />
'''Items to Discuss'''<br />
* Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!<br />
* Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)<br />
* [TE] Wrap-up notes<br />
<br />
=== August 10, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Python Wrapper and scripting<br />
** All changes merged into python-wrapper branch<br />
*** Tests pass<br />
*** Still need to update sub-branches<br />
** Minor fixes<br />
*** vtkPlot3DReader upgrade bug<br />
*** Added tuple exporting<br />
** Most of vtk examples can be exported [https://gist.github.com/rexissimus/5ac7374cde8525d84f9443f019d234e2 example]<br />
*** Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary<br />
**** Could be a bug in the old vistrail, but not sure<br />
* [DK] Subworkflows and uuids<br />
<br />
=== August 3, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Wrapper patches now stored as [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/vtk_patch.py python code]<br />
** Connection variables now lowercase<br />
** Can we use code as output variables?<br />
*** E.g. "vtk_dataset_mapper.GetOutput(0)"<br />
*** Could mutate output when output method is only supposed to be called once<br />
*** Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)<br />
<br />
=== July 27, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] DB bundle serialization implemented<br />
<br />
'''Items to Discuss'''<br />
* [TE] Finished using patch in script [https://github.com/VisTrails/VisTrails/tree/python-wrapper-script-patches python-wrapper-script-patches]<br />
** locale?<br />
** [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d new patching example] vs [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 old patching example]<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/parse.py#L135-L273 Example of new patches] <br />
** Legacy '.vtkInstance' is now added as a patch "self.vtkInstance=self" (see new patching example)<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/api.py New api for executing standalone modules]<br />
*** Accessing module was disabled for core/api, but a different api for this is probably better anyway.<br />
*** api.Package now gives [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d#file-new_patching-py-L26 directly callable module compute functions]<br />
*** Only works on modules not using interpreter functionality ('.interpreter', etc.)<br />
**** for zip file modules and others that need temp files and use the interpreter for this, need to modify the module, but could work around with python libraries for managing temp files<br />
** Fixed more renaming issues - [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/scripts.py#L63-L64 function argument names should not be replaced]<br />
** Still need to do something nicer for the patch formatting<br />
*** patch keys currently contain non-python-variable characters but this can be fixed<br />
*** Need to replace string template with python variables that is replacable using RedBaron<br />
**** Patches could then look like:<br />
<br />
def vtkInstance():<br />
self.vtkInstance = self<br />
<br />
=== July 20, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] New patch method for python-wrapper [https://gist.github.com/rexissimus/009b303e4cadcd4b8f2fec6cf848cb2b example]<br />
** A patch is a string template with self/input/output variables for patching a class method <br />
** Patches are stored in xml specification<br />
** Patches are assigned to methods in ClassSpec's<br />
** Used both for execution in vistrails and when exporting as script<br />
*** Executed with "exec" when run in vistrails<br />
*** Turns patches into natural calls in script<br />
** Wrapping VTK library no longer needed<br />
*** Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)<br />
** Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)<br />
** Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)<br />
** TODO's<br />
*** Add to export_as_script<br />
*** Patch Modules that represent Python functions as well<br />
<br />
* [vistrails-users] Read data from SQL example?<br />
<br />
=== July 13, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] bundles on sql-alchemy branch<br />
** Bundle, BundleMapping, and BundleSerializer<br />
** New BaseSerializers to deal with versions of directory, zip, db serializers<br />
** Can register the BundleSerializers with the base serializers<br />
** Translation between bundle versions<br />
** Bundle.add_object "just works" if an appropriate mapping was registered<br />
<br />
'''Items to Discuss'''<br />
* [TE] Simplified running module using API for scripting [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 example]<br />
** API Module can be executed using '''compute''' method<br />
*** Takes list of outputs to compute + inputs as keyword arguments<br />
* Status of general cell synchronization (vistrails-users)<br />
<br />
=== July 6, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] Working on bundles<br />
<br />
'''Items to Discuss'''<br />
* [TE] Using vistrails API in scripting [https://gist.github.com/rexissimus/034d9c4dedc3299a079e84981dedd8c7 example 1 (API with VTK)] [https://gist.github.com/rexissimus/a4a9ca6c08d4058c64967d2b35415fd5 example 2 (DownloadFile only)]<br />
** Call module in script using vistrails API<br />
** Can be used when no export method is defined.<br />
* [TE] [https://github.com/VisTrails/VisTrails/commit/c76c109349769fee1280f85c22678380a2f9cd39 Server news bug] Critical?<br />
**[RR] No, unreleased<br />
* [DK] New bundle organization<br />
** BundleMapping contains individual mappings from raw objects to BundleObj<br />
** Must be separate in order to have in-memory, separate from serializer settings<br />
<br />
=== June 29, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as python script<br />
** Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) [https://gist.github.com/rexissimus/16caf5200802e150296f62a0c7844c5a example]<br />
** Preludes (imports) still need improvements<br />
** Works with vtkInteractionHandler (clipping plane)<br />
** Resulting clipping plane is incorrect? (supposed to hide one side of the plane)<br />
** vtkRendererOutput Uses VTKRenderWindow as output<br />
*** Should support different output modes?<br />
*** Does not work with multiple outputs (Next cell is displayed when current is closed)<br />
*** Uses wrapped vtkInstance (as vvtk)<br />
**** Create new patch system with code that can be used both in vistrails and in a python script?<br />
<br />
=== June 22, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fix for upgraded analogies (PR [https://github.com/VisTrails/VisTrails/pull/1177 #1177])<br />
** Uses module location to find module remap (The best we can do right now?)<br />
** Also fixes remapped port names for connections to existing modules <br />
** May still fail on complex upgrades like adding a function on an upgraded module that has a different port name<br />
<br />
* [TE] Working on exporting wrapped libraries (VTK)<br />
<br />
=== June 15, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Added depth logic to python script exporter ([https://github.com/VisTrails/VisTrails/commit/50f220e4f6abe5a7bb8689f73f456f43df2dbc12 full commit])<br />
** Fully implements connection merging and list looping<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Making analogy work on upgraded workflows ([https://github.com/VisTrails/VisTrails/issues/1175 #1175])<br />
** Analogy only works We need to use the non-upgraded action chain<br />
** But pipelines need to be valid (matcher uses module portspecs)<br />
** Tried to get upgrade remap from upgrade action, but this seems impossible<br />
** Use module location instead?<br />
*** Location should be identical for one-to-one module upgrades<br />
<br />
=== June 1, 2016 ===<br />
<br />
'''Updates'''<br />
* VisTrails 2.2.4 announced (finally)<br />
** 5 survey replies<br />
** 47 downloads last week<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as script (python-wrapper)<br />
** to_python_script<br />
*** needs the pipeline (or at least the connections)<br />
**** Should not generate code for any possible VTK method (exporter detects which ports is used in the source)<br />
**** VTK needs to know connection order<br />
**** Adding functions directly to the script results in nicer code<br />
** Add support for multiple input connections<br />
** Add support for list looping<br />
<br />
=== May 25, 2016 ===<br />
<br />
'''Updates'''<br />
* Mailing-list fixed<br />
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])<br />
<br />
'''Items to Discuss'''<br />
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)<br />
* [TE] Union ports<br />
** Show union ports in pipeline (DEMO)<br />
<br />
=== May 18, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] mailing list still down - Julio working on it<br />
* [TE] Improved union port<br />
** New schema 1.0.5 adds portitem.union string attribute<br />
** Works with pipeline view, parameter exploration view, and mashup view<br />
** Should work well for matplotlib<br />
** Use with VTK ports?<br />
*** It uses [item_1,item_2] style ports usually with different tuple length.<br />
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]<br />
<br />
=== May 11, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?<br />
*** Shows single port for all ports with same sort_key<br />
*** Select specific type when adding/showing<br />
*** Only one type should be visible on the module at any time<br />
*** Making tuple parameters union types is probably too complicated<br />
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly) <br />
<br />
* [TE] Feature suggestion: Let connections connect to functions/parameters<br />
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters<br />
** Use separate views for ports and functions (like before)<br />
*** Allow rearranging of functions<br />
** Pros:<br />
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.<br />
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.<br />
**** Makes List module simpler<br />
*** Connect to parameter in tuple<br />
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds. <br />
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.<br />
** Cons:<br />
*** Interface more complicated and crowded<br />
*** More difficult for users to grasp?<br />
*** No immediate use case<br />
*** Makes no union type for tuple parameters a bigger problem.<br />
<br />
* [TE] Publish user survey?<br />
** No response from vistrails release yet<br />
** Not announced on vistrails-users yet.<br />
<br />
=== May 4, 2016 ===<br />
'''Updates'''<br />
* VisTrails 2.2.4 released<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
<br />
=== April 27, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Wrapping Bokeh<br />
** New method type "operation"<br />
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output<br />
**** Glyph sets function name and arguments as output<br />
***** Not serializable, but should always be computed together with the subject of the operation.<br />
**** Figure applies Glyph operation on itself<br />
*** Useful for matplotlib?<br />
**** matplotlib currently find actors by searching upstream?<br />
**** But this is more difficult since matplotlib is not function-based.<br />
** New "union" port type<br />
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value<br />
**** Uses only first value in tuple<br />
**** Bokeh have many properties that support (list, string, Float)<br />
*** Does not work with depth=1 types (e.g. Float of depth 1)<br />
**** Turn Depth 1 types with known type into lists?<br />
<br />
* [TE] New bugfix release<br />
** Finalize survey<br />
<br />
=== April 20, 2016 ===<br />
'''Updates'''<br />
* Merged upgrade fixes<br />
** Now has infinite loop test<br />
** loop test now correctly loops 5 times<br />
*** remap_module used wrong (current) package version when creating ModuleDescriptor<br />
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])<br />
<br />
'''Items to Discuss'''<br />
* New bugfix release<br />
** Merge usagestats<br />
** Need to finalize survey<br />
<br />
=== April 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?<br />
<br />
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])<br />
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]<br />
** Cherry-pick test to preserve source?<br />
** Need maxPipelineFixAttempts option?<br />
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.<br />
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?<br />
*** Would need to be moved into handle_invalid_pipeline<br />
<br />
=== April 6, 2016 ===<br />
'''Updates'''<br />
* [TE] Query View improvements<br />
** Done except for displaying "Version Info" and #1164<br />
* [RR] Working on finishing new interpreter (finally!)<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Try latest upgrade first, then previous.<br />
** If all fail, select latest upgrade?<br />
<br />
=== March 30, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Query View improvements<br />
** Unified do_version_switch and get_upgrades as validate_version()<br />
** Problems with query view using its own controller<br />
*** Fixed module info not being displayed for query results<br />
*** Still no version info for query results<br />
*** Still an issue with version query view not being updated<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== Older meetings ==<br />
* [[Development/2015]]<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=12442Development2016-09-14T13:27:55Z<p>Remi: /* September 14, 2016 */</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2016 ==<br />
=== September 14, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
=== August 31, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Went over vgc.poly.edu/vistrails.org config:<br />
** Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)<br />
** Fixed usersguide scripts (won't disappear again)<br />
** Moved some websites over to HTTPS<br />
** [https://docs.google.com/document/d/18YZewjR2YY9cQ9pzdA-nU9jbu2L7jYcPlDi3iaPadwA/edit Websites]<br />
* [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs ([http://vistrails.readthedocs.io/ currently there]) and users' guide under the same RTD "project"<br />
* [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that<br />
<br />
'''Items to Discuss'''<br />
* [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/<br />
** Get rid of more services? (svn, trac, ...)<br />
* [RR] Notebook execution works. How do we support more languages (than Python)?<br />
** Need: detection, metadata reader, code to input values<br />
* [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?<br />
<br />
=== August 17, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Fixed [https://github.com/VisTrails/VisTrails/commit/2cd42d156243067a7663bb0bf0156e5d87188c2f parameter order upgrade bug] that breaks VTK examples.<br />
* [TE] Matplotlib and sklearn are exportable using module-to-script api<br />
* [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.<br />
<br />
'''Items to Discuss'''<br />
* Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!<br />
* Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)<br />
* [TE] Wrap-up notes<br />
<br />
=== August 10, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Python Wrapper and scripting<br />
** All changes merged into python-wrapper branch<br />
*** Tests pass<br />
*** Still need to update sub-branches<br />
** Minor fixes<br />
*** vtkPlot3DReader upgrade bug<br />
*** Added tuple exporting<br />
** Most of vtk examples can be exported [https://gist.github.com/rexissimus/5ac7374cde8525d84f9443f019d234e2 example]<br />
*** Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary<br />
**** Could be a bug in the old vistrail, but not sure<br />
* [DK] Subworkflows and uuids<br />
<br />
=== August 3, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Wrapper patches now stored as [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/vtk_patch.py python code]<br />
** Connection variables now lowercase<br />
** Can we use code as output variables?<br />
*** E.g. "vtk_dataset_mapper.GetOutput(0)"<br />
*** Could mutate output when output method is only supposed to be called once<br />
*** Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)<br />
<br />
=== July 27, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] DB bundle serialization implemented<br />
<br />
'''Items to Discuss'''<br />
* [TE] Finished using patch in script [https://github.com/VisTrails/VisTrails/tree/python-wrapper-script-patches python-wrapper-script-patches]<br />
** locale?<br />
** [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d new patching example] vs [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 old patching example]<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/parse.py#L135-L273 Example of new patches] <br />
** Legacy '.vtkInstance' is now added as a patch "self.vtkInstance=self" (see new patching example)<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/api.py New api for executing standalone modules]<br />
*** Accessing module was disabled for core/api, but a different api for this is probably better anyway.<br />
*** api.Package now gives [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d#file-new_patching-py-L26 directly callable module compute functions]<br />
*** Only works on modules not using interpreter functionality ('.interpreter', etc.)<br />
**** for zip file modules and others that need temp files and use the interpreter for this, need to modify the module, but could work around with python libraries for managing temp files<br />
** Fixed more renaming issues - [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/scripts.py#L63-L64 function argument names should not be replaced]<br />
** Still need to do something nicer for the patch formatting<br />
*** patch keys currently contain non-python-variable characters but this can be fixed<br />
*** Need to replace string template with python variables that is replacable using RedBaron<br />
**** Patches could then look like:<br />
<br />
def vtkInstance():<br />
self.vtkInstance = self<br />
<br />
=== July 20, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] New patch method for python-wrapper [https://gist.github.com/rexissimus/009b303e4cadcd4b8f2fec6cf848cb2b example]<br />
** A patch is a string template with self/input/output variables for patching a class method <br />
** Patches are stored in xml specification<br />
** Patches are assigned to methods in ClassSpec's<br />
** Used both for execution in vistrails and when exporting as script<br />
*** Executed with "exec" when run in vistrails<br />
*** Turns patches into natural calls in script<br />
** Wrapping VTK library no longer needed<br />
*** Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)<br />
** Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)<br />
** Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)<br />
** TODO's<br />
*** Add to export_as_script<br />
*** Patch Modules that represent Python functions as well<br />
<br />
* [vistrails-users] Read data from SQL example?<br />
<br />
=== July 13, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] bundles on sql-alchemy branch<br />
** Bundle, BundleMapping, and BundleSerializer<br />
** New BaseSerializers to deal with versions of directory, zip, db serializers<br />
** Can register the BundleSerializers with the base serializers<br />
** Translation between bundle versions<br />
** Bundle.add_object "just works" if an appropriate mapping was registered<br />
<br />
'''Items to Discuss'''<br />
* [TE] Simplified running module using API for scripting [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 example]<br />
** API Module can be executed using '''compute''' method<br />
*** Takes list of outputs to compute + inputs as keyword arguments<br />
* Status of general cell synchronization (vistrails-users)<br />
<br />
=== July 6, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] Working on bundles<br />
<br />
'''Items to Discuss'''<br />
* [TE] Using vistrails API in scripting [https://gist.github.com/rexissimus/034d9c4dedc3299a079e84981dedd8c7 example 1 (API with VTK)] [https://gist.github.com/rexissimus/a4a9ca6c08d4058c64967d2b35415fd5 example 2 (DownloadFile only)]<br />
** Call module in script using vistrails API<br />
** Can be used when no export method is defined.<br />
* [TE] [https://github.com/VisTrails/VisTrails/commit/c76c109349769fee1280f85c22678380a2f9cd39 Server news bug] Critical?<br />
**[RR] No, unreleased<br />
* [DK] New bundle organization<br />
** BundleMapping contains individual mappings from raw objects to BundleObj<br />
** Must be separate in order to have in-memory, separate from serializer settings<br />
<br />
=== June 29, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as python script<br />
** Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) [https://gist.github.com/rexissimus/16caf5200802e150296f62a0c7844c5a example]<br />
** Preludes (imports) still need improvements<br />
** Works with vtkInteractionHandler (clipping plane)<br />
** Resulting clipping plane is incorrect? (supposed to hide one side of the plane)<br />
** vtkRendererOutput Uses VTKRenderWindow as output<br />
*** Should support different output modes?<br />
*** Does not work with multiple outputs (Next cell is displayed when current is closed)<br />
*** Uses wrapped vtkInstance (as vvtk)<br />
**** Create new patch system with code that can be used both in vistrails and in a python script?<br />
<br />
=== June 22, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fix for upgraded analogies (PR [https://github.com/VisTrails/VisTrails/pull/1177 #1177])<br />
** Uses module location to find module remap (The best we can do right now?)<br />
** Also fixes remapped port names for connections to existing modules <br />
** May still fail on complex upgrades like adding a function on an upgraded module that has a different port name<br />
<br />
* [TE] Working on exporting wrapped libraries (VTK)<br />
<br />
=== June 15, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Added depth logic to python script exporter ([https://github.com/VisTrails/VisTrails/commit/50f220e4f6abe5a7bb8689f73f456f43df2dbc12 full commit])<br />
** Fully implements connection merging and list looping<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Making analogy work on upgraded workflows ([https://github.com/VisTrails/VisTrails/issues/1175 #1175])<br />
** Analogy only works We need to use the non-upgraded action chain<br />
** But pipelines need to be valid (matcher uses module portspecs)<br />
** Tried to get upgrade remap from upgrade action, but this seems impossible<br />
** Use module location instead?<br />
*** Location should be identical for one-to-one module upgrades<br />
<br />
=== June 1, 2016 ===<br />
<br />
'''Updates'''<br />
* VisTrails 2.2.4 announced (finally)<br />
** 5 survey replies<br />
** 47 downloads last week<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as script (python-wrapper)<br />
** to_python_script<br />
*** needs the pipeline (or at least the connections)<br />
**** Should not generate code for any possible VTK method (exporter detects which ports is used in the source)<br />
**** VTK needs to know connection order<br />
**** Adding functions directly to the script results in nicer code<br />
** Add support for multiple input connections<br />
** Add support for list looping<br />
<br />
=== May 25, 2016 ===<br />
<br />
'''Updates'''<br />
* Mailing-list fixed<br />
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])<br />
<br />
'''Items to Discuss'''<br />
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)<br />
* [TE] Union ports<br />
** Show union ports in pipeline (DEMO)<br />
<br />
=== May 18, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] mailing list still down - Julio working on it<br />
* [TE] Improved union port<br />
** New schema 1.0.5 adds portitem.union string attribute<br />
** Works with pipeline view, parameter exploration view, and mashup view<br />
** Should work well for matplotlib<br />
** Use with VTK ports?<br />
*** It uses [item_1,item_2] style ports usually with different tuple length.<br />
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]<br />
<br />
=== May 11, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?<br />
*** Shows single port for all ports with same sort_key<br />
*** Select specific type when adding/showing<br />
*** Only one type should be visible on the module at any time<br />
*** Making tuple parameters union types is probably too complicated<br />
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly) <br />
<br />
* [TE] Feature suggestion: Let connections connect to functions/parameters<br />
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters<br />
** Use separate views for ports and functions (like before)<br />
*** Allow rearranging of functions<br />
** Pros:<br />
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.<br />
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.<br />
**** Makes List module simpler<br />
*** Connect to parameter in tuple<br />
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds. <br />
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.<br />
** Cons:<br />
*** Interface more complicated and crowded<br />
*** More difficult for users to grasp?<br />
*** No immediate use case<br />
*** Makes no union type for tuple parameters a bigger problem.<br />
<br />
* [TE] Publish user survey?<br />
** No response from vistrails release yet<br />
** Not announced on vistrails-users yet.<br />
<br />
=== May 4, 2016 ===<br />
'''Updates'''<br />
* VisTrails 2.2.4 released<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
<br />
=== April 27, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Wrapping Bokeh<br />
** New method type "operation"<br />
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output<br />
**** Glyph sets function name and arguments as output<br />
***** Not serializable, but should always be computed together with the subject of the operation.<br />
**** Figure applies Glyph operation on itself<br />
*** Useful for matplotlib?<br />
**** matplotlib currently find actors by searching upstream?<br />
**** But this is more difficult since matplotlib is not function-based.<br />
** New "union" port type<br />
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value<br />
**** Uses only first value in tuple<br />
**** Bokeh have many properties that support (list, string, Float)<br />
*** Does not work with depth=1 types (e.g. Float of depth 1)<br />
**** Turn Depth 1 types with known type into lists?<br />
<br />
* [TE] New bugfix release<br />
** Finalize survey<br />
<br />
=== April 20, 2016 ===<br />
'''Updates'''<br />
* Merged upgrade fixes<br />
** Now has infinite loop test<br />
** loop test now correctly loops 5 times<br />
*** remap_module used wrong (current) package version when creating ModuleDescriptor<br />
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])<br />
<br />
'''Items to Discuss'''<br />
* New bugfix release<br />
** Merge usagestats<br />
** Need to finalize survey<br />
<br />
=== April 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?<br />
<br />
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])<br />
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]<br />
** Cherry-pick test to preserve source?<br />
** Need maxPipelineFixAttempts option?<br />
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.<br />
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?<br />
*** Would need to be moved into handle_invalid_pipeline<br />
<br />
=== April 6, 2016 ===<br />
'''Updates'''<br />
* [TE] Query View improvements<br />
** Done except for displaying "Version Info" and #1164<br />
* [RR] Working on finishing new interpreter (finally!)<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Try latest upgrade first, then previous.<br />
** If all fail, select latest upgrade?<br />
<br />
=== March 30, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Query View improvements<br />
** Unified do_version_switch and get_upgrades as validate_version()<br />
** Problems with query view using its own controller<br />
*** Fixed module info not being displayed for query results<br />
*** Still no version info for query results<br />
*** Still an issue with version query view not being updated<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== Older meetings ==<br />
* [[Development/2015]]<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=12400Development2016-08-31T14:54:33Z<p>Remi: /* August 31, 2016 */</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2016 ==<br />
=== August 31, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Went over vgc.poly.edu/vistrails.org config:<br />
** Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)<br />
** Fixed usersguide scripts (won't disappear again)<br />
** Moved some websites over to HTTPS<br />
** [https://docs.google.com/document/d/18YZewjR2YY9cQ9pzdA-nU9jbu2L7jYcPlDi3iaPadwA/edit Websites]<br />
* [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs ([http://vistrails.readthedocs.io/ currently there]) and users' guide under the same RTD "project"<br />
<br />
'''Items to Discuss'''<br />
* [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/<br />
** Get rid of more services? (svn, trac, ...)<br />
* [RR] Notebook execution works. How do we support more languages (than Python)?<br />
** Need: detection, metadata reader, code to input values<br />
* [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?<br />
<br />
=== August 17, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Fixed [https://github.com/VisTrails/VisTrails/commit/2cd42d156243067a7663bb0bf0156e5d87188c2f parameter order upgrade bug] that breaks VTK examples.<br />
* [TE] Matplotlib and sklearn are exportable using module-to-script api<br />
* [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.<br />
<br />
'''Items to Discuss'''<br />
* Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!<br />
* Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)<br />
* [TE] Wrap-up notes<br />
<br />
=== August 10, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Python Wrapper and scripting<br />
** All changes merged into python-wrapper branch<br />
*** Tests pass<br />
*** Still need to update sub-branches<br />
** Minor fixes<br />
*** vtkPlot3DReader upgrade bug<br />
*** Added tuple exporting<br />
** Most of vtk examples can be exported [https://gist.github.com/rexissimus/5ac7374cde8525d84f9443f019d234e2 example]<br />
*** Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary<br />
**** Could be a bug in the old vistrail, but not sure<br />
* [DK] Subworkflows and uuids<br />
<br />
=== August 3, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Wrapper patches now stored as [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/vtk_patch.py python code]<br />
** Connection variables now lowercase<br />
** Can we use code as output variables?<br />
*** E.g. "vtk_dataset_mapper.GetOutput(0)"<br />
*** Could mutate output when output method is only supposed to be called once<br />
*** Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)<br />
<br />
=== July 27, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] DB bundle serialization implemented<br />
<br />
'''Items to Discuss'''<br />
* [TE] Finished using patch in script [https://github.com/VisTrails/VisTrails/tree/python-wrapper-script-patches python-wrapper-script-patches]<br />
** locale?<br />
** [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d new patching example] vs [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 old patching example]<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/parse.py#L135-L273 Example of new patches] <br />
** Legacy '.vtkInstance' is now added as a patch "self.vtkInstance=self" (see new patching example)<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/api.py New api for executing standalone modules]<br />
*** Accessing module was disabled for core/api, but a different api for this is probably better anyway.<br />
*** api.Package now gives [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d#file-new_patching-py-L26 directly callable module compute functions]<br />
*** Only works on modules not using interpreter functionality ('.interpreter', etc.)<br />
**** for zip file modules and others that need temp files and use the interpreter for this, need to modify the module, but could work around with python libraries for managing temp files<br />
** Fixed more renaming issues - [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/scripts.py#L63-L64 function argument names should not be replaced]<br />
** Still need to do something nicer for the patch formatting<br />
*** patch keys currently contain non-python-variable characters but this can be fixed<br />
*** Need to replace string template with python variables that is replacable using RedBaron<br />
**** Patches could then look like:<br />
<br />
def vtkInstance():<br />
self.vtkInstance = self<br />
<br />
=== July 20, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] New patch method for python-wrapper [https://gist.github.com/rexissimus/009b303e4cadcd4b8f2fec6cf848cb2b example]<br />
** A patch is a string template with self/input/output variables for patching a class method <br />
** Patches are stored in xml specification<br />
** Patches are assigned to methods in ClassSpec's<br />
** Used both for execution in vistrails and when exporting as script<br />
*** Executed with "exec" when run in vistrails<br />
*** Turns patches into natural calls in script<br />
** Wrapping VTK library no longer needed<br />
*** Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)<br />
** Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)<br />
** Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)<br />
** TODO's<br />
*** Add to export_as_script<br />
*** Patch Modules that represent Python functions as well<br />
<br />
* [vistrails-users] Read data from SQL example?<br />
<br />
=== July 13, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] bundles on sql-alchemy branch<br />
** Bundle, BundleMapping, and BundleSerializer<br />
** New BaseSerializers to deal with versions of directory, zip, db serializers<br />
** Can register the BundleSerializers with the base serializers<br />
** Translation between bundle versions<br />
** Bundle.add_object "just works" if an appropriate mapping was registered<br />
<br />
'''Items to Discuss'''<br />
* [TE] Simplified running module using API for scripting [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 example]<br />
** API Module can be executed using '''compute''' method<br />
*** Takes list of outputs to compute + inputs as keyword arguments<br />
* Status of general cell synchronization (vistrails-users)<br />
<br />
=== July 6, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] Working on bundles<br />
<br />
'''Items to Discuss'''<br />
* [TE] Using vistrails API in scripting [https://gist.github.com/rexissimus/034d9c4dedc3299a079e84981dedd8c7 example 1 (API with VTK)] [https://gist.github.com/rexissimus/a4a9ca6c08d4058c64967d2b35415fd5 example 2 (DownloadFile only)]<br />
** Call module in script using vistrails API<br />
** Can be used when no export method is defined.<br />
* [TE] [https://github.com/VisTrails/VisTrails/commit/c76c109349769fee1280f85c22678380a2f9cd39 Server news bug] Critical?<br />
**[RR] No, unreleased<br />
* [DK] New bundle organization<br />
** BundleMapping contains individual mappings from raw objects to BundleObj<br />
** Must be separate in order to have in-memory, separate from serializer settings<br />
<br />
=== June 29, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as python script<br />
** Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) [https://gist.github.com/rexissimus/16caf5200802e150296f62a0c7844c5a example]<br />
** Preludes (imports) still need improvements<br />
** Works with vtkInteractionHandler (clipping plane)<br />
** Resulting clipping plane is incorrect? (supposed to hide one side of the plane)<br />
** vtkRendererOutput Uses VTKRenderWindow as output<br />
*** Should support different output modes?<br />
*** Does not work with multiple outputs (Next cell is displayed when current is closed)<br />
*** Uses wrapped vtkInstance (as vvtk)<br />
**** Create new patch system with code that can be used both in vistrails and in a python script?<br />
<br />
=== June 22, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fix for upgraded analogies (PR [https://github.com/VisTrails/VisTrails/pull/1177 #1177])<br />
** Uses module location to find module remap (The best we can do right now?)<br />
** Also fixes remapped port names for connections to existing modules <br />
** May still fail on complex upgrades like adding a function on an upgraded module that has a different port name<br />
<br />
* [TE] Working on exporting wrapped libraries (VTK)<br />
<br />
=== June 15, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Added depth logic to python script exporter ([https://github.com/VisTrails/VisTrails/commit/50f220e4f6abe5a7bb8689f73f456f43df2dbc12 full commit])<br />
** Fully implements connection merging and list looping<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Making analogy work on upgraded workflows ([https://github.com/VisTrails/VisTrails/issues/1175 #1175])<br />
** Analogy only works We need to use the non-upgraded action chain<br />
** But pipelines need to be valid (matcher uses module portspecs)<br />
** Tried to get upgrade remap from upgrade action, but this seems impossible<br />
** Use module location instead?<br />
*** Location should be identical for one-to-one module upgrades<br />
<br />
=== June 1, 2016 ===<br />
<br />
'''Updates'''<br />
* VisTrails 2.2.4 announced (finally)<br />
** 5 survey replies<br />
** 47 downloads last week<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as script (python-wrapper)<br />
** to_python_script<br />
*** needs the pipeline (or at least the connections)<br />
**** Should not generate code for any possible VTK method (exporter detects which ports is used in the source)<br />
**** VTK needs to know connection order<br />
**** Adding functions directly to the script results in nicer code<br />
** Add support for multiple input connections<br />
** Add support for list looping<br />
<br />
=== May 25, 2016 ===<br />
<br />
'''Updates'''<br />
* Mailing-list fixed<br />
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])<br />
<br />
'''Items to Discuss'''<br />
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)<br />
* [TE] Union ports<br />
** Show union ports in pipeline (DEMO)<br />
<br />
=== May 18, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] mailing list still down - Julio working on it<br />
* [TE] Improved union port<br />
** New schema 1.0.5 adds portitem.union string attribute<br />
** Works with pipeline view, parameter exploration view, and mashup view<br />
** Should work well for matplotlib<br />
** Use with VTK ports?<br />
*** It uses [item_1,item_2] style ports usually with different tuple length.<br />
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]<br />
<br />
=== May 11, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?<br />
*** Shows single port for all ports with same sort_key<br />
*** Select specific type when adding/showing<br />
*** Only one type should be visible on the module at any time<br />
*** Making tuple parameters union types is probably too complicated<br />
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly) <br />
<br />
* [TE] Feature suggestion: Let connections connect to functions/parameters<br />
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters<br />
** Use separate views for ports and functions (like before)<br />
*** Allow rearranging of functions<br />
** Pros:<br />
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.<br />
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.<br />
**** Makes List module simpler<br />
*** Connect to parameter in tuple<br />
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds. <br />
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.<br />
** Cons:<br />
*** Interface more complicated and crowded<br />
*** More difficult for users to grasp?<br />
*** No immediate use case<br />
*** Makes no union type for tuple parameters a bigger problem.<br />
<br />
* [TE] Publish user survey?<br />
** No response from vistrails release yet<br />
** Not announced on vistrails-users yet.<br />
<br />
=== May 4, 2016 ===<br />
'''Updates'''<br />
* VisTrails 2.2.4 released<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
<br />
=== April 27, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Wrapping Bokeh<br />
** New method type "operation"<br />
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output<br />
**** Glyph sets function name and arguments as output<br />
***** Not serializable, but should always be computed together with the subject of the operation.<br />
**** Figure applies Glyph operation on itself<br />
*** Useful for matplotlib?<br />
**** matplotlib currently find actors by searching upstream?<br />
**** But this is more difficult since matplotlib is not function-based.<br />
** New "union" port type<br />
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value<br />
**** Uses only first value in tuple<br />
**** Bokeh have many properties that support (list, string, Float)<br />
*** Does not work with depth=1 types (e.g. Float of depth 1)<br />
**** Turn Depth 1 types with known type into lists?<br />
<br />
* [TE] New bugfix release<br />
** Finalize survey<br />
<br />
=== April 20, 2016 ===<br />
'''Updates'''<br />
* Merged upgrade fixes<br />
** Now has infinite loop test<br />
** loop test now correctly loops 5 times<br />
*** remap_module used wrong (current) package version when creating ModuleDescriptor<br />
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])<br />
<br />
'''Items to Discuss'''<br />
* New bugfix release<br />
** Merge usagestats<br />
** Need to finalize survey<br />
<br />
=== April 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?<br />
<br />
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])<br />
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]<br />
** Cherry-pick test to preserve source?<br />
** Need maxPipelineFixAttempts option?<br />
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.<br />
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?<br />
*** Would need to be moved into handle_invalid_pipeline<br />
<br />
=== April 6, 2016 ===<br />
'''Updates'''<br />
* [TE] Query View improvements<br />
** Done except for displaying "Version Info" and #1164<br />
* [RR] Working on finishing new interpreter (finally!)<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Try latest upgrade first, then previous.<br />
** If all fail, select latest upgrade?<br />
<br />
=== March 30, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Query View improvements<br />
** Unified do_version_switch and get_upgrades as validate_version()<br />
** Problems with query view using its own controller<br />
*** Fixed module info not being displayed for query results<br />
*** Still no version info for query results<br />
*** Still an issue with version query view not being updated<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== Older meetings ==<br />
* [[Development/2015]]<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=12399Development2016-08-31T14:50:48Z<p>Remi: /* August 31, 2016 */</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2016 ==<br />
=== August 31, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Went over vgc.poly.edu/vistrails.org config:<br />
** Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)<br />
** Fixed usersguide scripts (won't disappear again)<br />
** Moved some websites over to HTTPS<br />
** [https://docs.google.com/document/d/18YZewjR2YY9cQ9pzdA-nU9jbu2L7jYcPlDi3iaPadwA/edit Websites]<br />
* [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs ([http://vistrails.readthedocs.io/ currently there]) and users' guide under the same RTD "project"<br />
<br />
'''Items to Discuss'''<br />
* [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/<br />
** Get rid of more services? (svn, trac, ...)<br />
* [RR] Notebook execution works. How do we support more languages (than Python)?<br />
** Need: detection, metadata reader, code to input values<br />
<br />
=== August 17, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Fixed [https://github.com/VisTrails/VisTrails/commit/2cd42d156243067a7663bb0bf0156e5d87188c2f parameter order upgrade bug] that breaks VTK examples.<br />
* [TE] Matplotlib and sklearn are exportable using module-to-script api<br />
* [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.<br />
<br />
'''Items to Discuss'''<br />
* Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!<br />
* Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)<br />
* [TE] Wrap-up notes<br />
<br />
=== August 10, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Python Wrapper and scripting<br />
** All changes merged into python-wrapper branch<br />
*** Tests pass<br />
*** Still need to update sub-branches<br />
** Minor fixes<br />
*** vtkPlot3DReader upgrade bug<br />
*** Added tuple exporting<br />
** Most of vtk examples can be exported [https://gist.github.com/rexissimus/5ac7374cde8525d84f9443f019d234e2 example]<br />
*** Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary<br />
**** Could be a bug in the old vistrail, but not sure<br />
* [DK] Subworkflows and uuids<br />
<br />
=== August 3, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Wrapper patches now stored as [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/vtk_patch.py python code]<br />
** Connection variables now lowercase<br />
** Can we use code as output variables?<br />
*** E.g. "vtk_dataset_mapper.GetOutput(0)"<br />
*** Could mutate output when output method is only supposed to be called once<br />
*** Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)<br />
<br />
=== July 27, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] DB bundle serialization implemented<br />
<br />
'''Items to Discuss'''<br />
* [TE] Finished using patch in script [https://github.com/VisTrails/VisTrails/tree/python-wrapper-script-patches python-wrapper-script-patches]<br />
** locale?<br />
** [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d new patching example] vs [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 old patching example]<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/parse.py#L135-L273 Example of new patches] <br />
** Legacy '.vtkInstance' is now added as a patch "self.vtkInstance=self" (see new patching example)<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/api.py New api for executing standalone modules]<br />
*** Accessing module was disabled for core/api, but a different api for this is probably better anyway.<br />
*** api.Package now gives [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d#file-new_patching-py-L26 directly callable module compute functions]<br />
*** Only works on modules not using interpreter functionality ('.interpreter', etc.)<br />
**** for zip file modules and others that need temp files and use the interpreter for this, need to modify the module, but could work around with python libraries for managing temp files<br />
** Fixed more renaming issues - [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/scripts.py#L63-L64 function argument names should not be replaced]<br />
** Still need to do something nicer for the patch formatting<br />
*** patch keys currently contain non-python-variable characters but this can be fixed<br />
*** Need to replace string template with python variables that is replacable using RedBaron<br />
**** Patches could then look like:<br />
<br />
def vtkInstance():<br />
self.vtkInstance = self<br />
<br />
=== July 20, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] New patch method for python-wrapper [https://gist.github.com/rexissimus/009b303e4cadcd4b8f2fec6cf848cb2b example]<br />
** A patch is a string template with self/input/output variables for patching a class method <br />
** Patches are stored in xml specification<br />
** Patches are assigned to methods in ClassSpec's<br />
** Used both for execution in vistrails and when exporting as script<br />
*** Executed with "exec" when run in vistrails<br />
*** Turns patches into natural calls in script<br />
** Wrapping VTK library no longer needed<br />
*** Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)<br />
** Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)<br />
** Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)<br />
** TODO's<br />
*** Add to export_as_script<br />
*** Patch Modules that represent Python functions as well<br />
<br />
* [vistrails-users] Read data from SQL example?<br />
<br />
=== July 13, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] bundles on sql-alchemy branch<br />
** Bundle, BundleMapping, and BundleSerializer<br />
** New BaseSerializers to deal with versions of directory, zip, db serializers<br />
** Can register the BundleSerializers with the base serializers<br />
** Translation between bundle versions<br />
** Bundle.add_object "just works" if an appropriate mapping was registered<br />
<br />
'''Items to Discuss'''<br />
* [TE] Simplified running module using API for scripting [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 example]<br />
** API Module can be executed using '''compute''' method<br />
*** Takes list of outputs to compute + inputs as keyword arguments<br />
* Status of general cell synchronization (vistrails-users)<br />
<br />
=== July 6, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] Working on bundles<br />
<br />
'''Items to Discuss'''<br />
* [TE] Using vistrails API in scripting [https://gist.github.com/rexissimus/034d9c4dedc3299a079e84981dedd8c7 example 1 (API with VTK)] [https://gist.github.com/rexissimus/a4a9ca6c08d4058c64967d2b35415fd5 example 2 (DownloadFile only)]<br />
** Call module in script using vistrails API<br />
** Can be used when no export method is defined.<br />
* [TE] [https://github.com/VisTrails/VisTrails/commit/c76c109349769fee1280f85c22678380a2f9cd39 Server news bug] Critical?<br />
**[RR] No, unreleased<br />
* [DK] New bundle organization<br />
** BundleMapping contains individual mappings from raw objects to BundleObj<br />
** Must be separate in order to have in-memory, separate from serializer settings<br />
<br />
=== June 29, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as python script<br />
** Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) [https://gist.github.com/rexissimus/16caf5200802e150296f62a0c7844c5a example]<br />
** Preludes (imports) still need improvements<br />
** Works with vtkInteractionHandler (clipping plane)<br />
** Resulting clipping plane is incorrect? (supposed to hide one side of the plane)<br />
** vtkRendererOutput Uses VTKRenderWindow as output<br />
*** Should support different output modes?<br />
*** Does not work with multiple outputs (Next cell is displayed when current is closed)<br />
*** Uses wrapped vtkInstance (as vvtk)<br />
**** Create new patch system with code that can be used both in vistrails and in a python script?<br />
<br />
=== June 22, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fix for upgraded analogies (PR [https://github.com/VisTrails/VisTrails/pull/1177 #1177])<br />
** Uses module location to find module remap (The best we can do right now?)<br />
** Also fixes remapped port names for connections to existing modules <br />
** May still fail on complex upgrades like adding a function on an upgraded module that has a different port name<br />
<br />
* [TE] Working on exporting wrapped libraries (VTK)<br />
<br />
=== June 15, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Added depth logic to python script exporter ([https://github.com/VisTrails/VisTrails/commit/50f220e4f6abe5a7bb8689f73f456f43df2dbc12 full commit])<br />
** Fully implements connection merging and list looping<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Making analogy work on upgraded workflows ([https://github.com/VisTrails/VisTrails/issues/1175 #1175])<br />
** Analogy only works We need to use the non-upgraded action chain<br />
** But pipelines need to be valid (matcher uses module portspecs)<br />
** Tried to get upgrade remap from upgrade action, but this seems impossible<br />
** Use module location instead?<br />
*** Location should be identical for one-to-one module upgrades<br />
<br />
=== June 1, 2016 ===<br />
<br />
'''Updates'''<br />
* VisTrails 2.2.4 announced (finally)<br />
** 5 survey replies<br />
** 47 downloads last week<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as script (python-wrapper)<br />
** to_python_script<br />
*** needs the pipeline (or at least the connections)<br />
**** Should not generate code for any possible VTK method (exporter detects which ports is used in the source)<br />
**** VTK needs to know connection order<br />
**** Adding functions directly to the script results in nicer code<br />
** Add support for multiple input connections<br />
** Add support for list looping<br />
<br />
=== May 25, 2016 ===<br />
<br />
'''Updates'''<br />
* Mailing-list fixed<br />
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])<br />
<br />
'''Items to Discuss'''<br />
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)<br />
* [TE] Union ports<br />
** Show union ports in pipeline (DEMO)<br />
<br />
=== May 18, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] mailing list still down - Julio working on it<br />
* [TE] Improved union port<br />
** New schema 1.0.5 adds portitem.union string attribute<br />
** Works with pipeline view, parameter exploration view, and mashup view<br />
** Should work well for matplotlib<br />
** Use with VTK ports?<br />
*** It uses [item_1,item_2] style ports usually with different tuple length.<br />
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]<br />
<br />
=== May 11, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?<br />
*** Shows single port for all ports with same sort_key<br />
*** Select specific type when adding/showing<br />
*** Only one type should be visible on the module at any time<br />
*** Making tuple parameters union types is probably too complicated<br />
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly) <br />
<br />
* [TE] Feature suggestion: Let connections connect to functions/parameters<br />
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters<br />
** Use separate views for ports and functions (like before)<br />
*** Allow rearranging of functions<br />
** Pros:<br />
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.<br />
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.<br />
**** Makes List module simpler<br />
*** Connect to parameter in tuple<br />
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds. <br />
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.<br />
** Cons:<br />
*** Interface more complicated and crowded<br />
*** More difficult for users to grasp?<br />
*** No immediate use case<br />
*** Makes no union type for tuple parameters a bigger problem.<br />
<br />
* [TE] Publish user survey?<br />
** No response from vistrails release yet<br />
** Not announced on vistrails-users yet.<br />
<br />
=== May 4, 2016 ===<br />
'''Updates'''<br />
* VisTrails 2.2.4 released<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
<br />
=== April 27, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Wrapping Bokeh<br />
** New method type "operation"<br />
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output<br />
**** Glyph sets function name and arguments as output<br />
***** Not serializable, but should always be computed together with the subject of the operation.<br />
**** Figure applies Glyph operation on itself<br />
*** Useful for matplotlib?<br />
**** matplotlib currently find actors by searching upstream?<br />
**** But this is more difficult since matplotlib is not function-based.<br />
** New "union" port type<br />
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value<br />
**** Uses only first value in tuple<br />
**** Bokeh have many properties that support (list, string, Float)<br />
*** Does not work with depth=1 types (e.g. Float of depth 1)<br />
**** Turn Depth 1 types with known type into lists?<br />
<br />
* [TE] New bugfix release<br />
** Finalize survey<br />
<br />
=== April 20, 2016 ===<br />
'''Updates'''<br />
* Merged upgrade fixes<br />
** Now has infinite loop test<br />
** loop test now correctly loops 5 times<br />
*** remap_module used wrong (current) package version when creating ModuleDescriptor<br />
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])<br />
<br />
'''Items to Discuss'''<br />
* New bugfix release<br />
** Merge usagestats<br />
** Need to finalize survey<br />
<br />
=== April 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?<br />
<br />
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])<br />
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]<br />
** Cherry-pick test to preserve source?<br />
** Need maxPipelineFixAttempts option?<br />
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.<br />
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?<br />
*** Would need to be moved into handle_invalid_pipeline<br />
<br />
=== April 6, 2016 ===<br />
'''Updates'''<br />
* [TE] Query View improvements<br />
** Done except for displaying "Version Info" and #1164<br />
* [RR] Working on finishing new interpreter (finally!)<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Try latest upgrade first, then previous.<br />
** If all fail, select latest upgrade?<br />
<br />
=== March 30, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Query View improvements<br />
** Unified do_version_switch and get_upgrades as validate_version()<br />
** Problems with query view using its own controller<br />
*** Fixed module info not being displayed for query results<br />
*** Still no version info for query results<br />
*** Still an issue with version query view not being updated<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== Older meetings ==<br />
* [[Development/2015]]<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=12398Development2016-08-31T14:45:00Z<p>Remi: /* August 31, 2016 */</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2016 ==<br />
=== August 31, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Went over vgc.poly.edu/vistrails.org config:<br />
** Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)<br />
** Fixed usersguide scripts (won't disappear again)<br />
** Moved some websites over to HTTPS<br />
** [https://docs.google.com/document/d/18YZewjR2YY9cQ9pzdA-nU9jbu2L7jYcPlDi3iaPadwA/edit Websites]<br />
<br />
'''Items to Discuss'''<br />
* [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/<br />
** Get rid of more services? (svn, trac, ...)<br />
* [RR] Notebook execution works. How do we support more languages (than Python)?<br />
** Need: detection, metadata reader, code to input values<br />
<br />
=== August 17, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Fixed [https://github.com/VisTrails/VisTrails/commit/2cd42d156243067a7663bb0bf0156e5d87188c2f parameter order upgrade bug] that breaks VTK examples.<br />
* [TE] Matplotlib and sklearn are exportable using module-to-script api<br />
* [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.<br />
<br />
'''Items to Discuss'''<br />
* Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!<br />
* Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)<br />
* [TE] Wrap-up notes<br />
<br />
=== August 10, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Python Wrapper and scripting<br />
** All changes merged into python-wrapper branch<br />
*** Tests pass<br />
*** Still need to update sub-branches<br />
** Minor fixes<br />
*** vtkPlot3DReader upgrade bug<br />
*** Added tuple exporting<br />
** Most of vtk examples can be exported [https://gist.github.com/rexissimus/5ac7374cde8525d84f9443f019d234e2 example]<br />
*** Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary<br />
**** Could be a bug in the old vistrail, but not sure<br />
* [DK] Subworkflows and uuids<br />
<br />
=== August 3, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Wrapper patches now stored as [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/vtk_patch.py python code]<br />
** Connection variables now lowercase<br />
** Can we use code as output variables?<br />
*** E.g. "vtk_dataset_mapper.GetOutput(0)"<br />
*** Could mutate output when output method is only supposed to be called once<br />
*** Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)<br />
<br />
=== July 27, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] DB bundle serialization implemented<br />
<br />
'''Items to Discuss'''<br />
* [TE] Finished using patch in script [https://github.com/VisTrails/VisTrails/tree/python-wrapper-script-patches python-wrapper-script-patches]<br />
** locale?<br />
** [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d new patching example] vs [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 old patching example]<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/parse.py#L135-L273 Example of new patches] <br />
** Legacy '.vtkInstance' is now added as a patch "self.vtkInstance=self" (see new patching example)<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/api.py New api for executing standalone modules]<br />
*** Accessing module was disabled for core/api, but a different api for this is probably better anyway.<br />
*** api.Package now gives [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d#file-new_patching-py-L26 directly callable module compute functions]<br />
*** Only works on modules not using interpreter functionality ('.interpreter', etc.)<br />
**** for zip file modules and others that need temp files and use the interpreter for this, need to modify the module, but could work around with python libraries for managing temp files<br />
** Fixed more renaming issues - [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/scripts.py#L63-L64 function argument names should not be replaced]<br />
** Still need to do something nicer for the patch formatting<br />
*** patch keys currently contain non-python-variable characters but this can be fixed<br />
*** Need to replace string template with python variables that is replacable using RedBaron<br />
**** Patches could then look like:<br />
<br />
def vtkInstance():<br />
self.vtkInstance = self<br />
<br />
=== July 20, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] New patch method for python-wrapper [https://gist.github.com/rexissimus/009b303e4cadcd4b8f2fec6cf848cb2b example]<br />
** A patch is a string template with self/input/output variables for patching a class method <br />
** Patches are stored in xml specification<br />
** Patches are assigned to methods in ClassSpec's<br />
** Used both for execution in vistrails and when exporting as script<br />
*** Executed with "exec" when run in vistrails<br />
*** Turns patches into natural calls in script<br />
** Wrapping VTK library no longer needed<br />
*** Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)<br />
** Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)<br />
** Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)<br />
** TODO's<br />
*** Add to export_as_script<br />
*** Patch Modules that represent Python functions as well<br />
<br />
* [vistrails-users] Read data from SQL example?<br />
<br />
=== July 13, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] bundles on sql-alchemy branch<br />
** Bundle, BundleMapping, and BundleSerializer<br />
** New BaseSerializers to deal with versions of directory, zip, db serializers<br />
** Can register the BundleSerializers with the base serializers<br />
** Translation between bundle versions<br />
** Bundle.add_object "just works" if an appropriate mapping was registered<br />
<br />
'''Items to Discuss'''<br />
* [TE] Simplified running module using API for scripting [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 example]<br />
** API Module can be executed using '''compute''' method<br />
*** Takes list of outputs to compute + inputs as keyword arguments<br />
* Status of general cell synchronization (vistrails-users)<br />
<br />
=== July 6, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] Working on bundles<br />
<br />
'''Items to Discuss'''<br />
* [TE] Using vistrails API in scripting [https://gist.github.com/rexissimus/034d9c4dedc3299a079e84981dedd8c7 example 1 (API with VTK)] [https://gist.github.com/rexissimus/a4a9ca6c08d4058c64967d2b35415fd5 example 2 (DownloadFile only)]<br />
** Call module in script using vistrails API<br />
** Can be used when no export method is defined.<br />
* [TE] [https://github.com/VisTrails/VisTrails/commit/c76c109349769fee1280f85c22678380a2f9cd39 Server news bug] Critical?<br />
**[RR] No, unreleased<br />
* [DK] New bundle organization<br />
** BundleMapping contains individual mappings from raw objects to BundleObj<br />
** Must be separate in order to have in-memory, separate from serializer settings<br />
<br />
=== June 29, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as python script<br />
** Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) [https://gist.github.com/rexissimus/16caf5200802e150296f62a0c7844c5a example]<br />
** Preludes (imports) still need improvements<br />
** Works with vtkInteractionHandler (clipping plane)<br />
** Resulting clipping plane is incorrect? (supposed to hide one side of the plane)<br />
** vtkRendererOutput Uses VTKRenderWindow as output<br />
*** Should support different output modes?<br />
*** Does not work with multiple outputs (Next cell is displayed when current is closed)<br />
*** Uses wrapped vtkInstance (as vvtk)<br />
**** Create new patch system with code that can be used both in vistrails and in a python script?<br />
<br />
=== June 22, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fix for upgraded analogies (PR [https://github.com/VisTrails/VisTrails/pull/1177 #1177])<br />
** Uses module location to find module remap (The best we can do right now?)<br />
** Also fixes remapped port names for connections to existing modules <br />
** May still fail on complex upgrades like adding a function on an upgraded module that has a different port name<br />
<br />
* [TE] Working on exporting wrapped libraries (VTK)<br />
<br />
=== June 15, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Added depth logic to python script exporter ([https://github.com/VisTrails/VisTrails/commit/50f220e4f6abe5a7bb8689f73f456f43df2dbc12 full commit])<br />
** Fully implements connection merging and list looping<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Making analogy work on upgraded workflows ([https://github.com/VisTrails/VisTrails/issues/1175 #1175])<br />
** Analogy only works We need to use the non-upgraded action chain<br />
** But pipelines need to be valid (matcher uses module portspecs)<br />
** Tried to get upgrade remap from upgrade action, but this seems impossible<br />
** Use module location instead?<br />
*** Location should be identical for one-to-one module upgrades<br />
<br />
=== June 1, 2016 ===<br />
<br />
'''Updates'''<br />
* VisTrails 2.2.4 announced (finally)<br />
** 5 survey replies<br />
** 47 downloads last week<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as script (python-wrapper)<br />
** to_python_script<br />
*** needs the pipeline (or at least the connections)<br />
**** Should not generate code for any possible VTK method (exporter detects which ports is used in the source)<br />
**** VTK needs to know connection order<br />
**** Adding functions directly to the script results in nicer code<br />
** Add support for multiple input connections<br />
** Add support for list looping<br />
<br />
=== May 25, 2016 ===<br />
<br />
'''Updates'''<br />
* Mailing-list fixed<br />
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])<br />
<br />
'''Items to Discuss'''<br />
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)<br />
* [TE] Union ports<br />
** Show union ports in pipeline (DEMO)<br />
<br />
=== May 18, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] mailing list still down - Julio working on it<br />
* [TE] Improved union port<br />
** New schema 1.0.5 adds portitem.union string attribute<br />
** Works with pipeline view, parameter exploration view, and mashup view<br />
** Should work well for matplotlib<br />
** Use with VTK ports?<br />
*** It uses [item_1,item_2] style ports usually with different tuple length.<br />
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]<br />
<br />
=== May 11, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?<br />
*** Shows single port for all ports with same sort_key<br />
*** Select specific type when adding/showing<br />
*** Only one type should be visible on the module at any time<br />
*** Making tuple parameters union types is probably too complicated<br />
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly) <br />
<br />
* [TE] Feature suggestion: Let connections connect to functions/parameters<br />
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters<br />
** Use separate views for ports and functions (like before)<br />
*** Allow rearranging of functions<br />
** Pros:<br />
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.<br />
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.<br />
**** Makes List module simpler<br />
*** Connect to parameter in tuple<br />
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds. <br />
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.<br />
** Cons:<br />
*** Interface more complicated and crowded<br />
*** More difficult for users to grasp?<br />
*** No immediate use case<br />
*** Makes no union type for tuple parameters a bigger problem.<br />
<br />
* [TE] Publish user survey?<br />
** No response from vistrails release yet<br />
** Not announced on vistrails-users yet.<br />
<br />
=== May 4, 2016 ===<br />
'''Updates'''<br />
* VisTrails 2.2.4 released<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
<br />
=== April 27, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Wrapping Bokeh<br />
** New method type "operation"<br />
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output<br />
**** Glyph sets function name and arguments as output<br />
***** Not serializable, but should always be computed together with the subject of the operation.<br />
**** Figure applies Glyph operation on itself<br />
*** Useful for matplotlib?<br />
**** matplotlib currently find actors by searching upstream?<br />
**** But this is more difficult since matplotlib is not function-based.<br />
** New "union" port type<br />
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value<br />
**** Uses only first value in tuple<br />
**** Bokeh have many properties that support (list, string, Float)<br />
*** Does not work with depth=1 types (e.g. Float of depth 1)<br />
**** Turn Depth 1 types with known type into lists?<br />
<br />
* [TE] New bugfix release<br />
** Finalize survey<br />
<br />
=== April 20, 2016 ===<br />
'''Updates'''<br />
* Merged upgrade fixes<br />
** Now has infinite loop test<br />
** loop test now correctly loops 5 times<br />
*** remap_module used wrong (current) package version when creating ModuleDescriptor<br />
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])<br />
<br />
'''Items to Discuss'''<br />
* New bugfix release<br />
** Merge usagestats<br />
** Need to finalize survey<br />
<br />
=== April 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?<br />
<br />
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])<br />
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]<br />
** Cherry-pick test to preserve source?<br />
** Need maxPipelineFixAttempts option?<br />
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.<br />
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?<br />
*** Would need to be moved into handle_invalid_pipeline<br />
<br />
=== April 6, 2016 ===<br />
'''Updates'''<br />
* [TE] Query View improvements<br />
** Done except for displaying "Version Info" and #1164<br />
* [RR] Working on finishing new interpreter (finally!)<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Try latest upgrade first, then previous.<br />
** If all fail, select latest upgrade?<br />
<br />
=== March 30, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Query View improvements<br />
** Unified do_version_switch and get_upgrades as validate_version()<br />
** Problems with query view using its own controller<br />
*** Fixed module info not being displayed for query results<br />
*** Still no version info for query results<br />
*** Still an issue with version query view not being updated<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== Older meetings ==<br />
* [[Development/2015]]<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=12396Development2016-08-30T21:14:02Z<p>Remi: Move 2015 meetings to separate page</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2016 ==<br />
=== August 31, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Went over vgc.poly.edu/vistrails.org config:<br />
** Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)<br />
** Fixed usersguide scripts (won't disappear again)<br />
** Moved some websites over to HTTPS<br />
** [https://docs.google.com/document/d/18YZewjR2YY9cQ9pzdA-nU9jbu2L7jYcPlDi3iaPadwA/edit Websites]<br />
<br />
'''Items to Discuss'''<br />
* [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/<br />
** Get rid of more services? (svn, trac, ...)<br />
<br />
=== August 17, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Fixed [https://github.com/VisTrails/VisTrails/commit/2cd42d156243067a7663bb0bf0156e5d87188c2f parameter order upgrade bug] that breaks VTK examples.<br />
* [TE] Matplotlib and sklearn are exportable using module-to-script api<br />
* [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.<br />
<br />
'''Items to Discuss'''<br />
* Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!<br />
* Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)<br />
* [TE] Wrap-up notes<br />
<br />
=== August 10, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Python Wrapper and scripting<br />
** All changes merged into python-wrapper branch<br />
*** Tests pass<br />
*** Still need to update sub-branches<br />
** Minor fixes<br />
*** vtkPlot3DReader upgrade bug<br />
*** Added tuple exporting<br />
** Most of vtk examples can be exported [https://gist.github.com/rexissimus/5ac7374cde8525d84f9443f019d234e2 example]<br />
*** Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary<br />
**** Could be a bug in the old vistrail, but not sure<br />
* [DK] Subworkflows and uuids<br />
<br />
=== August 3, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Wrapper patches now stored as [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/vtk_patch.py python code]<br />
** Connection variables now lowercase<br />
** Can we use code as output variables?<br />
*** E.g. "vtk_dataset_mapper.GetOutput(0)"<br />
*** Could mutate output when output method is only supposed to be called once<br />
*** Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)<br />
<br />
=== July 27, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] DB bundle serialization implemented<br />
<br />
'''Items to Discuss'''<br />
* [TE] Finished using patch in script [https://github.com/VisTrails/VisTrails/tree/python-wrapper-script-patches python-wrapper-script-patches]<br />
** locale?<br />
** [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d new patching example] vs [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 old patching example]<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/parse.py#L135-L273 Example of new patches] <br />
** Legacy '.vtkInstance' is now added as a patch "self.vtkInstance=self" (see new patching example)<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/api.py New api for executing standalone modules]<br />
*** Accessing module was disabled for core/api, but a different api for this is probably better anyway.<br />
*** api.Package now gives [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d#file-new_patching-py-L26 directly callable module compute functions]<br />
*** Only works on modules not using interpreter functionality ('.interpreter', etc.)<br />
**** for zip file modules and others that need temp files and use the interpreter for this, need to modify the module, but could work around with python libraries for managing temp files<br />
** Fixed more renaming issues - [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/scripts.py#L63-L64 function argument names should not be replaced]<br />
** Still need to do something nicer for the patch formatting<br />
*** patch keys currently contain non-python-variable characters but this can be fixed<br />
*** Need to replace string template with python variables that is replacable using RedBaron<br />
**** Patches could then look like:<br />
<br />
def vtkInstance():<br />
self.vtkInstance = self<br />
<br />
=== July 20, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] New patch method for python-wrapper [https://gist.github.com/rexissimus/009b303e4cadcd4b8f2fec6cf848cb2b example]<br />
** A patch is a string template with self/input/output variables for patching a class method <br />
** Patches are stored in xml specification<br />
** Patches are assigned to methods in ClassSpec's<br />
** Used both for execution in vistrails and when exporting as script<br />
*** Executed with "exec" when run in vistrails<br />
*** Turns patches into natural calls in script<br />
** Wrapping VTK library no longer needed<br />
*** Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)<br />
** Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)<br />
** Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)<br />
** TODO's<br />
*** Add to export_as_script<br />
*** Patch Modules that represent Python functions as well<br />
<br />
* [vistrails-users] Read data from SQL example?<br />
<br />
=== July 13, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] bundles on sql-alchemy branch<br />
** Bundle, BundleMapping, and BundleSerializer<br />
** New BaseSerializers to deal with versions of directory, zip, db serializers<br />
** Can register the BundleSerializers with the base serializers<br />
** Translation between bundle versions<br />
** Bundle.add_object "just works" if an appropriate mapping was registered<br />
<br />
'''Items to Discuss'''<br />
* [TE] Simplified running module using API for scripting [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 example]<br />
** API Module can be executed using '''compute''' method<br />
*** Takes list of outputs to compute + inputs as keyword arguments<br />
* Status of general cell synchronization (vistrails-users)<br />
<br />
=== July 6, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] Working on bundles<br />
<br />
'''Items to Discuss'''<br />
* [TE] Using vistrails API in scripting [https://gist.github.com/rexissimus/034d9c4dedc3299a079e84981dedd8c7 example 1 (API with VTK)] [https://gist.github.com/rexissimus/a4a9ca6c08d4058c64967d2b35415fd5 example 2 (DownloadFile only)]<br />
** Call module in script using vistrails API<br />
** Can be used when no export method is defined.<br />
* [TE] [https://github.com/VisTrails/VisTrails/commit/c76c109349769fee1280f85c22678380a2f9cd39 Server news bug] Critical?<br />
**[RR] No, unreleased<br />
* [DK] New bundle organization<br />
** BundleMapping contains individual mappings from raw objects to BundleObj<br />
** Must be separate in order to have in-memory, separate from serializer settings<br />
<br />
=== June 29, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as python script<br />
** Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) [https://gist.github.com/rexissimus/16caf5200802e150296f62a0c7844c5a example]<br />
** Preludes (imports) still need improvements<br />
** Works with vtkInteractionHandler (clipping plane)<br />
** Resulting clipping plane is incorrect? (supposed to hide one side of the plane)<br />
** vtkRendererOutput Uses VTKRenderWindow as output<br />
*** Should support different output modes?<br />
*** Does not work with multiple outputs (Next cell is displayed when current is closed)<br />
*** Uses wrapped vtkInstance (as vvtk)<br />
**** Create new patch system with code that can be used both in vistrails and in a python script?<br />
<br />
=== June 22, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fix for upgraded analogies (PR [https://github.com/VisTrails/VisTrails/pull/1177 #1177])<br />
** Uses module location to find module remap (The best we can do right now?)<br />
** Also fixes remapped port names for connections to existing modules <br />
** May still fail on complex upgrades like adding a function on an upgraded module that has a different port name<br />
<br />
* [TE] Working on exporting wrapped libraries (VTK)<br />
<br />
=== June 15, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Added depth logic to python script exporter ([https://github.com/VisTrails/VisTrails/commit/50f220e4f6abe5a7bb8689f73f456f43df2dbc12 full commit])<br />
** Fully implements connection merging and list looping<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Making analogy work on upgraded workflows ([https://github.com/VisTrails/VisTrails/issues/1175 #1175])<br />
** Analogy only works We need to use the non-upgraded action chain<br />
** But pipelines need to be valid (matcher uses module portspecs)<br />
** Tried to get upgrade remap from upgrade action, but this seems impossible<br />
** Use module location instead?<br />
*** Location should be identical for one-to-one module upgrades<br />
<br />
=== June 1, 2016 ===<br />
<br />
'''Updates'''<br />
* VisTrails 2.2.4 announced (finally)<br />
** 5 survey replies<br />
** 47 downloads last week<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as script (python-wrapper)<br />
** to_python_script<br />
*** needs the pipeline (or at least the connections)<br />
**** Should not generate code for any possible VTK method (exporter detects which ports is used in the source)<br />
**** VTK needs to know connection order<br />
**** Adding functions directly to the script results in nicer code<br />
** Add support for multiple input connections<br />
** Add support for list looping<br />
<br />
=== May 25, 2016 ===<br />
<br />
'''Updates'''<br />
* Mailing-list fixed<br />
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])<br />
<br />
'''Items to Discuss'''<br />
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)<br />
* [TE] Union ports<br />
** Show union ports in pipeline (DEMO)<br />
<br />
=== May 18, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] mailing list still down - Julio working on it<br />
* [TE] Improved union port<br />
** New schema 1.0.5 adds portitem.union string attribute<br />
** Works with pipeline view, parameter exploration view, and mashup view<br />
** Should work well for matplotlib<br />
** Use with VTK ports?<br />
*** It uses [item_1,item_2] style ports usually with different tuple length.<br />
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]<br />
<br />
=== May 11, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?<br />
*** Shows single port for all ports with same sort_key<br />
*** Select specific type when adding/showing<br />
*** Only one type should be visible on the module at any time<br />
*** Making tuple parameters union types is probably too complicated<br />
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly) <br />
<br />
* [TE] Feature suggestion: Let connections connect to functions/parameters<br />
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters<br />
** Use separate views for ports and functions (like before)<br />
*** Allow rearranging of functions<br />
** Pros:<br />
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.<br />
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.<br />
**** Makes List module simpler<br />
*** Connect to parameter in tuple<br />
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds. <br />
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.<br />
** Cons:<br />
*** Interface more complicated and crowded<br />
*** More difficult for users to grasp?<br />
*** No immediate use case<br />
*** Makes no union type for tuple parameters a bigger problem.<br />
<br />
* [TE] Publish user survey?<br />
** No response from vistrails release yet<br />
** Not announced on vistrails-users yet.<br />
<br />
=== May 4, 2016 ===<br />
'''Updates'''<br />
* VisTrails 2.2.4 released<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
<br />
=== April 27, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Wrapping Bokeh<br />
** New method type "operation"<br />
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output<br />
**** Glyph sets function name and arguments as output<br />
***** Not serializable, but should always be computed together with the subject of the operation.<br />
**** Figure applies Glyph operation on itself<br />
*** Useful for matplotlib?<br />
**** matplotlib currently find actors by searching upstream?<br />
**** But this is more difficult since matplotlib is not function-based.<br />
** New "union" port type<br />
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value<br />
**** Uses only first value in tuple<br />
**** Bokeh have many properties that support (list, string, Float)<br />
*** Does not work with depth=1 types (e.g. Float of depth 1)<br />
**** Turn Depth 1 types with known type into lists?<br />
<br />
* [TE] New bugfix release<br />
** Finalize survey<br />
<br />
=== April 20, 2016 ===<br />
'''Updates'''<br />
* Merged upgrade fixes<br />
** Now has infinite loop test<br />
** loop test now correctly loops 5 times<br />
*** remap_module used wrong (current) package version when creating ModuleDescriptor<br />
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])<br />
<br />
'''Items to Discuss'''<br />
* New bugfix release<br />
** Merge usagestats<br />
** Need to finalize survey<br />
<br />
=== April 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?<br />
<br />
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])<br />
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]<br />
** Cherry-pick test to preserve source?<br />
** Need maxPipelineFixAttempts option?<br />
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.<br />
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?<br />
*** Would need to be moved into handle_invalid_pipeline<br />
<br />
=== April 6, 2016 ===<br />
'''Updates'''<br />
* [TE] Query View improvements<br />
** Done except for displaying "Version Info" and #1164<br />
* [RR] Working on finishing new interpreter (finally!)<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Try latest upgrade first, then previous.<br />
** If all fail, select latest upgrade?<br />
<br />
=== March 30, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Query View improvements<br />
** Unified do_version_switch and get_upgrades as validate_version()<br />
** Problems with query view using its own controller<br />
*** Fixed module info not being displayed for query results<br />
*** Still no version info for query results<br />
*** Still an issue with version query view not being updated<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== Older meetings ==<br />
* [[Development/2015]]<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=Development/2015&diff=12395Development/20152016-08-30T21:13:34Z<p>Remi: Created page with '=== December 30, 2015 === '''Updates''' * Bokeh package finished ** Can output to Browser/IPython/Spreadsheet (Limited functionality in spreadsheet) '''Items to Discuss''' * Upda…'</p>
<hr />
<div>=== December 30, 2015 ===<br />
'''Updates'''<br />
* Bokeh package finished<br />
** Can output to Browser/IPython/Spreadsheet (Limited functionality in spreadsheet)<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Bokeh needs QWebEngine (Chromium) in Qt5 which requires PyQt5<br />
*** Many plotting libraries only work in browsers<br />
*** But Chromium may be a big and controversial dependency<br />
*** Or create a browser version of the spreadsheet?<br />
** Some users have requested Python 3<br />
** Not backwards compatible?<br />
** Target for VisTrails 3.0 with new interpreter?<br />
<br />
=== December 16, 2015 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Users list question<br />
* [TE] Wrapping Bokeh<br />
** Added bokeh property parser<br />
*** complements numpydoc parser<br />
*** Not all attributes are available, partially because I am using a pre-release version.<br />
** Wrapped modules<br />
*** bokeh.plotting finished (Backwards like matplotlib: Create plot then draw on it)<br />
*** bokeh.charts mostly done (Missing arguments)<br />
** QWebView mostly works (but no zooming or buttons)<br />
*** It would be better to use external browser or QWebEngineView<br />
** Added improvements to wrapper<br />
*** Generalized port translations (Color/Path to native types)<br />
*** Can use multiple docstring/property parsers simultaneously<br />
<br />
=== December 9, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Looking into wrapper<br />
** Will port TensorFlow<br />
** Doing a pass on the wrapper code & doc<br />
*** UX improvements<br />
** Factorize high-level docstring parsers into core.wrapper? (sphinx, numpy, google)<br />
** Problem with name vs module_name (fails sklearn tests because class_by_name() is broken)<br />
** Looking into compressing specs, lazy registration of modules<br />
<br />
=== December 2, 2015 ===<br />
'''Updates'''<br />
* [TE] Package wrapping<br />
** Wrapped most of numpy/scipy using __all__<br />
** Added class attribute and method access for classes (Can add to class or to separate inspector module)<br />
** TODO: Document wrapping procedure<br />
<br />
'''Items to Discuss'''<br />
<br />
=== November 25, 2015 ===<br />
'''Updates'''<br />
* [RR] New interpreter is coming along<br />
** Everything is a stream<br />
** Task system, eventually work-stealing parallelism<br />
** Streams no longer advance in lock-step<br />
*** Allows for constructions like filter, join, sorted-merge<br />
** depth>1 still to be tested, no current plan to allow this through the module interface<br />
*** Through looping though, can do streams of streams<br />
<br />
'''Items to Discuss'''<br />
* [TE] Package wrapping<br />
** Wrapped most numpy functions (no polynomials)<br />
** and half of scipy classes and functions<br />
** Added functions manually from documentation (no parseable lists?)<br />
** Focusing on data flow constructs (no property/method access for classes)<br />
*** Using spec diff (empty for now) and dynamic parser (stores spec in .vistrails/)<br />
** Python class/function wrapper now stable<br />
** Add google doc parser?<br />
*** Is there a parser available?<br />
<br />
=== November 18, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Package wrapping<br />
** Numpy class wrapping works<br />
*** Basic type is List, since most types are array_likes.<br />
*** TODO: More classes and functions<br />
*** Some operations are in-place and some aren't, and docstring not super clear<br />
*** [RR] It seems only methods mutate, and most have a numpy.xx function equivalent; just manually go through the methods and remove the mutating ones?<br />
** Class wrapper is modular<br />
*** Docstring parser and type string parser can be customized<br />
*** Classes can have optional attribute/method ports<br />
**** Inspectors and attribute/method modules can be created separately<br />
** Use PythonCalc as an example<br />
*** Can be re-implemented as a function with a parseable docstring<br />
*** [RR] PythonCalc is just an example, and it might actually make sense to get rid of it, or do a proper math package with scalar operations (as separate modules, no combobox)<br />
** [RR] TensorFlow to use wrapping as well<br />
*** Very simple (only types 'tensor' & 'variable')<br />
<br />
=== November 11, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Package wrapping<br />
** Wrapping numpy's classes using numpydoc (ndarray)<br />
*** Wrapping constructor arguments, attribute getters/setters as module ports<br />
**** Then how to access attributes afterwards?<br />
***** Use input value port to class modules?<br />
*** Wrap methods as modules? E.g., `ndarray.shape`.<br />
*** Function wrapping is almost a subset of class wrapping<br />
**** May be able to use the same parsing/execution methods<br />
** VTK's non-getter/setter methods could be wrapped as modules<br />
*** We could then remove extra logic in interpreter for keeping function order<br />
** Bokeh uses autogenerated docstrings<br />
*** We may be able to read the specification directly<br />
<br />
* [RR] [https://github.com/VisTrails/VisTrails/pull/1138 TensorFlow package]<br />
** Basic setup working, can execute the Mandelbrot example<br />
** Will autogenerate the operations<br />
<br />
=== November 4, 2015 ===<br />
'''Updates'''<br />
* [RR] Interpreter work requires a fix for [https://github.com/VisTrails/VisTrails/issues/1137 DB issue #1137]<br />
* [DK] Kitware's Resonant [http://www.kitware.com/source/home/post/173]<br />
** [https://github.com/girder/girder Girder]: data management system<br />
** [https://github.com/Kitware/romanesco Romanesco]: execution engine, uses [http://www.celeryproject.org/ Celery] for task management<br />
** Resonant Flow: web app for editing and executing workflows<br />
<br />
'''Items to Discuss'''<br />
* [TE] Added OSX Lion (10.7) VM on build machine using vagrant (seems ok with license?)<br />
** New builds works on Lion<br />
** Change the minimum requirement to 10.7?<br />
* [TE] Package wrapping<br />
* Added upgrade suggestions using 2 spec versions ([http://pastebin.com/30XUU8Gp example])<br />
** using name edit distance to find matches<br />
** Could use something better like port similarity for modules, and type similarity for ports<br />
** Should write python upgrade code?<br />
** Added spec to sklearn package<br />
*** Can now diff spec versions and keep the spec static<br />
*** There will be problems with downgrades and package versions<br />
<br />
=== October 29, 2015 ===<br />
'''Updates'''<br />
* [TE] nightly-build-mac fixed (Needed to approve xcode license)<br />
<br />
'''Items to Discuss'''<br />
* [TE] matplotlib wrapper<br />
** ported to general spec<br />
** Diff can be used on general spec<br />
*** Only needed minor changes<br />
*** Fixed indexing bug that corrupted specs<br />
*** Can now move diff tools to core/wrapper<br />
** Add new plots?<br />
*** New plots in mpl 1.3: eventplot<br />
*** New plots in mpl 1.4: angle_spectrum, magnitude_spectrum, phase_spectrum', violinplot<br />
*** No new plots in mpl 1.5 (from looking at boilerplate.py)<br />
<br />
=== October 21, 2015 ===<br />
'''Updates'''<br />
* [RR] Alexis has arrived, will be working with [RR] on a more efficient interpreter<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1133 New matplotlib package]<br />
** Supporting multiple package versions<br />
*** Check which version can be loaded [https://github.com/VisTrails/VisTrails/pull/1135 #1135]<br />
*** We cannot show version requirements in the list of packages because old packages are loaded when importing the codepath<br />
** Visual diff does not work well, but we can use the spec differ to see differences.<br />
** Unifying vtk and matplotlib wrapping specs (WIP)<br />
*** And porting matplotlibs spec differ to work on the general spec<br />
*** Spec differ can then be used on vtk and other packages in the future <br />
*** Will enable automatic upgrade generation<br />
* [RR] Rework interpreter<br />
** restore abstraction between interpreter/module code<br />
** build looping, streaming, caching into the interpreter<br />
** rework cache<br />
** look into parallelism once basic functionality is there<br />
<br />
=== October 14, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1133 New matplotlib package] (adding numpydoc parser)<br />
<br />
=== October 7, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://github.com/VisTrails/VisTrails/tree/fix-matplotlib-parser New matplotlib parser] (adding numpydoc parser)<br />
** Added numpydoc attribute parser for plots.<br />
*** TODO: Need better port spec reconciliation with call signature parser.<br />
*** We can create a general numpydoc parser, but attribute types are unique to matplotlib.<br />
** Needs package versioning<br />
*** New spec will not support matplotlib < v1.4 due to changed path to axes classes.<br />
*** Load spec version corresponding to installed matplotlib version?<br />
*** Need version downgrades?<br />
<br />
=== September 30, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Updating Matplotlib parser<br />
** Matplotlib docstring parser fails on numpy docstrings<br />
** I have [https://github.com/VisTrails/VisTrails/commit/eae8b0c16260810587e47c41cab53552cbb3cf0a added basic numpydoc parsing]<br />
*** Only used by a few docstrings so far<br />
*** Parsing uses many sources (class tables, signatures, ACCEPTS, method docstrings, definition parsing).<br />
** Caswell said they were thinking about moving to traitlets, but this is not ready yet<br />
<br />
=== September 23, 2015 ===<br />
'''Updates'''<br />
* [TE] VisTrails 2.2.3 released<br />
** Also have pushed to PyPI, binstar, etc.<br />
** Sourceforge vs. GitHub<br />
*** should be able to host releases on GitHub<br />
*** nightly binaries pushed to sf each night<br />
*** old binaries?<br />
* [TE] Re-wrapping MatPlotlib<br />
** Keep static generation - Docstrings are brittle beween matplotlib versions<br />
** Keep generating the executable classes - but create functions, not vistrails Modules <br />
*** Why is there so much patching?<br />
** Re-implemented Alternate PortSpec for InputPortSpec<br />
*** It will now inherit specs from parent (No need to reimplement)<br />
** Module upgrades?<br />
<br />
'''Items to Discuss'''<br />
<br />
=== September 16, 2015 ===<br />
'''Updates'''<br />
* [TE] Job Monitor tests and documentation done<br />
** OK to leave document package in packages directory?<br />
<br />
'''Items to Discuss'''<br />
* [TE] Library wrapping: How to do code patching? [https://docs.google.com/drawings/d/1Sh3eOXot6v2y880jHKQrqXqAwr-0aoqYligg3z8G8MY/edit?usp=sharing Wrapping diagram]<br />
** Insert code into module template (Old VTK code)<br />
** Patch the library that is called (New VTK code)<br />
** Somehow store code in spec and apply when executed?<br />
*** Executed code needs at a minimum access to inputs, outputs, current class<br />
*** Use standard input/output dict and obj reference that the code operates on?<br />
*** Can this be done while keeping the execution abstract?<br />
<br />
* Do new release now?<br />
<br />
=== September 9, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* New release?<br />
** [RR] wants reprounzip<br />
** tej changes & doc<br />
** job monitor & jobmixin fixes<br />
** mongodb<br />
** reprounzip<br />
** warning: MplFigure type is int<br />
** tabledata: add Reshape, DictoToTable & ListToTable, fix ListToTable with numpy arrays<br />
** PythonSource can have same name for an input & output<br />
** don't show spreadsheet at exit<br />
<br />
=== September 2, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Limit autosaves? [https://github.com/VisTrails/VisTrails/pull/1126 #1126]<br />
<br />
=== August 26, 2015 ===<br />
'''Updates'''<br />
* [TE] Added Job support to Parameter Explorations (Requested by Colin), and Mashups<br />
** specify job ids, need to specify different ids for parameter explorations since they have the same version id<br />
** how to deal with parameters passed in on command line<br />
* [General] provenance: should be creating a new version when we execute workflow with changed parameters?<br />
** currently, custom_params annotation stores this in provenance currently<br />
** [RR] API doesn't record provenance if passing in parameters: http://git.io/vsAA5<br />
'''Items to Discuss'''<br />
<br />
=== August 19, 2015 ===<br />
'''Updates'''<br />
* [RR] Improvements to tej, users' guide entry & example [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
* [RR] Internal docs [https://github.com/VisTrails/VisTrails/pull/1116 #1116]<br />
** On ReadTheDocs: [http://vistrails.readthedocs.org/ vistrails.readthedocs.org]<br />
* [TE] Working on Job Monitor documentation<br />
<br />
'''Items to Discuss'''<br />
* [RR] JobMixin and JobMonitor: stable now?<br />
** Definitely needs more tests<br />
* [TE] Stop testing VisTrails 2.0?<br />
<br />
=== August 12, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Internal docs [https://github.com/VisTrails/VisTrails/pull/1116 #1116]<br />
** On ReadTheDocs: [http://vistrails.readthedocs.org/ vistrails.readthedocs.org]<br />
* [RR] JobMixin and JobMonitor: stable now?<br />
** Definitely needs more tests<br />
* [RR] Improvements to tej, users' guide entry & example [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
* [RR] Build broken on Travis; because of IPython 4 released today? ([https://travis-ci.org/VisTrails/VisTrails/builds/75279795 build 992]; [https://github.com/VisTrails/VisTrails/issues/1123 #1123])<br />
<br />
=== August 5, 2015 ===<br />
'''Updates'''<br />
* [TE] Fixes to Jobs<br />
** Could not delete jobs<br />
** Could not run job in group<br />
** Job not reset when calling ModuleSuspended<br />
** Added deleting job from context menu<br />
* [RR] Writing documentation for everything [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
<br />
'''Items to Discuss'''<br />
<br />
=== July 29, 2015 ===<br />
'''Updates'''<br />
* RR still looking into new interpreter thing<br />
** Goal is to take out scheduling logic from Module so it can be split in multiple processes, and so that smarter strategies can be added in time<br />
** This means some work on packages<br />
** Spreadsheet can live in kernel process? Still some UI stuff that will take work (changing configuration, persistent archive's viewer, ...)<br />
* [TE] Problems running examples [https://github.com/VisTrails/VisTrails/pull/1111 #1111]<br />
** Testing of more examples requires additional packages on the test machines.<br />
** Fixed faulty line-ending in PythonSource:s failing on Python 2.6.<br />
** Test suite now testing SUDSWebServices (If web service is down, test suite will fail)<br />
** preferences.py test failed reloading 'dialogs' package, switched to using 'URL'.<br />
<br />
'''Items to Discuss'''<br />
<br />
=== July 22, 2015 ===<br />
'''Updates'''<br />
* [TE] mailing lists back online<br />
* [TE] Working on [https://github.com/VisTrails/VisTrails/issues/1107 #1107]<br />
* [RR] Working on ReproZip package <br />
<br />
'''Items to Discuss'''<br />
* [RR] MongoDB package [https://github.com/VisTrails/VisTrails/pull/1106 #1106]<br />
* [RR] Example for tej docs? [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
<br />
=== July 15, 2015 ===<br />
'''Updates'''<br />
* 2.2.2 released<br />
<br />
'''Items to Discuss'''<br />
* RR is considering executing everything in an IPython kernel (i.e. separate Python interpreter, like the one spawned for a notebook)<br />
** It's a separate process that we can restart without restarting the app/gui<br />
** We can isolate the execution environment (e.g. for the server)<br />
** We can run the whole thing remotely (if your data is elsewhere, just run VisTrails locally and the pipeline elsewhere)<br />
** Pipeline execution no longer makes the interface hang, it just makes the kernel hang (but that's fine)<br />
** We can use notebooks as modules (probably way nicer than the PythonSource module)<br />
** We can run multiple kernels so long as the ports carry things that are serializable<br />
*** meaning we can put the multithreaded-interpreter without all the hacky parts it has now<br />
*** we can run IPython kernels in all the languages IPython supports, [https://github.com/ipython/ipython/wiki/IPython-kernels-for-other-languages currently 46]<br />
<br />
=== July 8, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Buildbot github hook not working after IP address change<br />
* [TE] No reference to VistrailsApplication ([https://github.com/VisTrails/VisTrails/issues/1103 #1103])<br />
** [RR] Added PR [https://github.com/VisTrails/VisTrails/pull/1104 #1104]<br />
* [TE] Reopening VT file after saving with bundled subworkflow won't offer subworkflow upgrade ([https://github.com/VisTrails/VisTrails/issues/1102 #1102])<br />
** allow manual delete to fix right now<br />
** fix this on top of the use-uuid branch<br />
* [TE] Release VisTrails 2.2.1? ([https://github.com/VisTrails/VisTrails/blob/v2.2/CHANGELOG CHANGELOG])<br />
** [https://github.com/VisTrails/VisTrails/pull/1093 output-modules-use-global-dir]<br />
** [https://github.com/VisTrails/VisTrails/milestones/version%202.2 Current issues for 2.2]<br />
<br />
=== July 1, 2015 ===<br />
'''Updates'''<br />
* [TE] PROV fixed<br />
* [TE] Working on subworkflow issues<br />
* [RR] Considering reworking the controller (log vs exception problem, retained upgrades causing interferences) and also the interpreter (IPython?)<br />
<br />
'''Items to Discuss'''<br />
<br />
=== June 24, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* UV-CDAT<br />
* [TE] Can a cyclic workflow be valid? ([https://github.com/VisTrails/VisTrails/issues/1097 #1097])<br />
** focus on disabling the ability to create cyclic pipelines because more things break than just this with a cyclic pipeline<br />
* [RR] Relative paths ([https://github.com/VisTrails/VisTrails/pull/1057 #1057])<br />
** This interacts with the new bundle; how do handle packing files inside the VT bundle?<br />
<br />
=== June 17, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] current_version and reusing existing upgrades are broken ([https://github.com/VisTrails/VisTrails/issues/1095 ticket #1095])<br />
** Could be that export to PROV is using an unflushed upgrade pipeline?<br />
** current_version would then be correct.<br />
** It may work to flush the actions before exporting?<br />
<br />
=== June 10, 2015 ===<br />
'''Updates'''<br />
* DAT: fixed VTK issue on Linux and Mac<br />
* Still crashes on Windows. Need help! Reminder: this works in the VisTrails spreadsheet (QCellPresenter), although no widget get changed there during a drag<br />
** Is it a VTK bug?<br />
** Is it simply impossible to change widgets during the drag, should we do it a different way?<br />
** Did I miss something that is done in VisTrails but somehow not in DAT?<br />
** Low prio, UV-CDAT doesn't run on Windows anyway<br />
<br />
'''Items to Discuss'''<br />
* [TE] Executable Paper ([https://github.com/VisTrails/VisTrails/issues/1086 ticket #1088] [https://github.com/VisTrails/VisTrails/pull/1088 Updated pull request])<br />
* Single instance code and batch mode<br />
<br />
=== June 3, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Executable Paper ([https://github.com/VisTrails/VisTrails/issues/1086 ticket #1088] [https://github.com/VisTrails/VisTrails/pull/1088 pr])<br />
** Requires fixes to command line parameters, Output modules, and batch mode<br />
** How to test this<br />
** Updated missing/outdated flags<br />
** Fixed view issues when generating graphs<br />
* [TE] batch mode<br />
** SpreadsheetOutput not enabled in batch mode. Should we check is_running_gui instead?<br />
** Other instance setting flags from caller<br />
** Is graphsAsPdf replacing spreadsheetDumpPdf?<br />
** graphsAsPdf true by Default?<br />
** Batch mode executing by default (Not needed when generating graphs)<br />
** Re-added workflowInfo as withWorkflowInfo for writing graph and xml workflow<br />
** [DK] batch mode should be outputting to files or stdout, shouldn't always trigger SpreadsheetMode<br />
** execute flag, maybe make execution the default and allow a "--no-execute" if you only want to capture graphs, for example<br />
<br />
=== May 27, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] API question<br />
** http://lists.vistrails.org/pipermail/vistrails-users/2015-May/000357.html<br />
* Status of DAT<br />
** Can add VCS plot to DAT but need to work on configuration windows<br />
** Documentation for graphics templates for UV-CDAT/vcs?<br />
* Qt support?<br />
<br />
=== May 20, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Reviving the DAT, integrating scripting & porting UV-CDAT!<br />
** On [https://github.com/VisTrails/DAT GitHub] ([https://github.com/VisTrails/DAT/issues issues])<br />
** Merging 2 years of development taking longer than expected, but getting there. The plan is to get the patches in VisTrails and never fork again, we never want to get in UV-CDAT's situation (and don't need to).<br />
** VTK cell works fine on Linux but there was flickering on Mac & Windows before; still issues on Mac (Windows status unknown)<br />
** Can get a VCS plot soon (but will need VTK cell fix)<br />
** Integrate in UV-CDAT's build system (so we have cdms, VCS, ...) -> RR can do this, low priority<br />
** How do we integrate scripting?<br />
*** We want to be able to seamlessly make changes to a plot by changing Python code<br />
*** Define new plots by entering Python code without writing modules/packages?<br />
* [RR] UV-CDAT [https://github.com/UV-CDAT/uvcdat/issues/1275 needs] [https://github.com/VisTrails/VisTrails/pull/1073 #1073], please take a look<br />
* [RR] Subworkflow issues<br />
** [https://github.com/VisTrails/VisTrails/issues/1065 #1065]: missing attribute (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/1066 #1066]: groups in subworkflows were ignored when finding dependencies (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/725 #725]: "upgrade subworkflow" button not popping up (fixed); also, namespace issue<br />
** [https://github.com/VisTrails/VisTrails/issues/1071 #1071]: subworkflows don't go through upgrades<br />
* [RR] [https://github.com/VisTrails/VisTrails/issues/1074 #1074]: can't load and edit a single pipeline if upgrade happen<br />
<br />
=== May 13, 2015 ===<br />
'''Updates'''<br />
* [RR] Test skipping whitelist ([https://github.com/VisTrails/VisTrails/pull/1069 #1069]) -- low priority<br />
* [RR] Custom matplotlib modules can't be compatible with both 2.1 and 2.2 ([https://github.com/VisTrails/VisTrails/issues/1067 #1067]); should be fixed for ALPS ([https://github.com/VisTrails/VisTrails/issues/1070 #1070])<br />
<br />
'''Items to Discuss'''<br />
* [RR] UV-CDAT [https://github.com/UV-CDAT/uvcdat/issues/1275 needs] [https://github.com/VisTrails/VisTrails/pull/1073 #1073], please take a look<br />
* [RR] Subworkflow issues<br />
** [https://github.com/VisTrails/VisTrails/issues/1065 #1065]: missing attribute (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/1066 #1066]: groups in subworkflows were ignored when finding dependencies (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/725 #725]: "upgrade subworkflow" button not popping up (fixed); also, namespace issue<br />
** [https://github.com/VisTrails/VisTrails/issues/1071 #1071]: subworkflows don't go through upgrades<br />
* [RR] [https://github.com/VisTrails/VisTrails/issues/1074 #1074]: can't load and edit a single pipeline if upgrade happen<br />
<br />
=== May 7, 2015 ===<br />
'''Updates'''<br />
* [RR] Export/import workflow to Python working!<br />
<br />
'''Items to Discuss'''<br />
* [Claudio] UV-CDAT<br />
** The UV-CDAT project is the biggest user base of VisTrails<br />
** VisTrails package management provides a lot of friction towards people plugging in their code<br />
** Need to make it easy to integrate your random Python scripts in the system without having to deal with all the boilerplate, at least in the first step<br />
** [RR] argues that modules are still good; UV-CDAT shouldn't move towards a purely script-based backend<br />
** [RR] export/import with Python could reduce a lot of that friction by allowing 1) to edit workflow as Python 2) to open up boxes automatically if needed code doesn't match actual modules<br />
** ...<br />
<br />
=== April 29, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] <strike>Abstractions</strike> subworkflows status ([https://github.com/VisTrails/VisTrails/labels/subworkflows tickets])<br />
* [RR] matplotlib compatibility (2.1 & 2.2), [https://github.com/VisTrails/VisTrails/issues/1067 #1067]<br />
** RR to try and fix ALPS matplotlib modules<br />
<br />
=== April 22, 2015 ===<br />
'''Updates'''<br />
* 2.2.0 has been released!<br />
* Windows issue (via email from Ryan)<br />
** issue with manifest file (may be a new file in VTK6?)<br />
** Tommy has regenerated new Windows builds<br />
* Binaries, pypi, and conda released<br />
* [RR] Export as script<br />
* Python sources using VTK need to switch to SetInputData (users should be aware of this)<br />
<br />
=== April 15, 2015 ===<br />
'''Updates'''<br />
* Ready for 2.2.0 (apart from binary/deps issues)<br />
** Missing some libs (scikit-learn, tej, tdparser, SQLAlchemy+connectors<br />
** Windows: runvistrails.py is no longer used, so the PATH is wrong<br />
** Windows: pip is broken, but it probably wouldn't work anyway because of permission issues (disable this?)<br />
* Queries, upgrades and getPipeline() usage ([https://github.com/VisTrails/VisTrails/issues/1054 #1054])<br />
** Getting a pipeline with getPipeline() is not safe: it might return an invalid pipeline<br />
** This is used in many places throughout the code, like queries<br />
** Upgrading would require going through the controller, but that creates new actions<br />
* [TC] Avoid copying a module's output if it's used as input by exactly one downstream module ([https://github.com/VisTrails/VisTrails/issues/1060 #1060]) (useful for big numpy arrays you can update in-place)<br />
<br />
=== April 8, 2015 ===<br />
'''Updates'''<br />
* [DK] Merged RR's changes for output modules (1012 and 1013)<br />
** RR will merge remaining changes, then create v2.2 branch!<br />
<br />
'''Items to Discuss'''<br />
* BNL need numpy array to VTK image<br />
** Looks like VTK has some helpers for this<br />
** We will help if issues arise<br />
** Will contribute back to VisTrails package<br />
* Upgrade issue: [https://github.com/VisTrails/VisTrails/issues/1017 #1017]<br />
** Automatic upgrades should happen between versions of provided upgrades<br />
** Our existing upgrades work around this so it doesn't need to be 2.2.0<br />
* Corner-case VTK modules<br />
** No longer need VTKCell input port, so don't interfere with registry and API anymore<br />
** Still work weirdly, people probably shouldn't use them<br />
** But we have lots of clunky modules since we wrap the whole of VTK; some people might rely on this and know how to use them, let's keep them anyway<br />
** Ready for 2.2.0<br />
<br />
=== April 1, 2015 ===<br />
'''Updates'''<br />
* [RR] UV-CDAT: bugfixing for 2.2, long-term plans: implement scripting import/export in VisTrails, port to UV-CDAT<br />
* Possibly, try to move to regular VisTrails to use new features<br />
<br />
'''Items to Discuss'''<br />
* 2.2 release: nothing much is pending anymore, release next week?<br />
** [TE] vtkExporter classes [https://github.com/VisTrails/VisTrails/issues/1032 #1032]<br />
** [RR] Hiding upgrades in version tree might make it (or might be 2.2.1 so we can test it out) [https://github.com/VisTrails/VisTrails/issues/949 #949]<br />
** Output modules changes to go in<br />
* add note to documentation about order of parameters in VTK<br />
* add issue about exporter upgrades if not already there<br />
<br />
=== March 25, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* 2.2 release<br />
** Checklist on Github: https://github.com/VisTrails/VisTrails/wiki/2.2-Release-Checklist<br />
** Review output modules<br />
** RR has a couple more issues to fix<br />
** Ready to go -- sign app package for OSX? [https://github.com/VisTrails/VisTrails/issues/984 #984] We need access to the Apple Membership team<br />
<br />
=== March 18, 2015 ===<br />
'''Updates'''<br />
* [TE] VTK6 works<br />
<br />
'''Items to Discuss'''<br />
* [RR] Release v2.1.5 with backported tabledata?<br />
** MTA example needs updated tabledata (for JoinTables)<br />
** Google Maps package still not available<br />
* [RR] Work torward v2.2.0?<br />
** changes:<br />
*** new persistence<br />
*** API changes<br />
*** output module changes (upgrades?), maintain cells but try to upgrade<br />
*** not wrapping stuff<br />
*** VTK6? yes<br />
*** JobSubmission stuff?<br />
*** relabeling for upgrades [https://github.com/VisTrails/VisTrails/issues/949 #949]<br />
** makes sense, needs the tree view code to be updated, check selection<br />
** See [https://github.com/VisTrails/VisTrails/wiki/2.2-Release-Checklist 2.2 checklist]<br />
* Discussion of [https://github.com/VisTrails/VisTrails/issues/1016 #1016]<br />
** plumbing between outputs and output modes, how to define a mode that works for many outputs without writing for each output?<br />
<br />
=== March 11, 2015 ===<br />
'''Updates'''<br />
* [RR] persistent_archive done; merge? ([https://github.com/VisTrails/VisTrails/pull/755 #755])<br />
** note about the focus events for widgets<br />
** TE be aware of file_archive for future binaries that include persistent_archive<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/998 New VTK package] finished<br />
<br />
'''Items to Discuss'''<br />
* [RR] Unmark UV-CDAT/VisTrails as a fork of VisTrails/VisTrails? (see last week; decision needed)<br />
** RR email to JF about this<br />
** yes; email sent to Github<br />
* [RR] Switching order of output ports ([https://github.com/VisTrails/VisTrails/pull/1006 #1006])<br />
** added port specs are sorted at a separate spot (Module.*_port_specs properties) than those in the registry (which are sorted in the registry), but those two lists are just combined without respect to sort keys<br />
** need to determine whether the two lists should be merged or remain distinct<br />
** should make sure that order of input ports and output ports makes parallel connections for things with same order<br />
** DK suggests breaking backward compatibility here: workflows still run, can fix easily if a problem in an existing package.<br />
* [RR] Question about output modes ([https://github.com/VisTrails/VisTrails/pull/1007 #1007]), how to integrate in API ([https://github.com/VisTrails/VisTrails/pull/24 #24])<br />
** Should ImageFileMode be removed? ("image" is not a mode, "file" is)<br />
** ImageOutput missing?<br />
** Feel free to change how formats works<br />
* [TE] Test suite segfaults on Fedora 17 virtualbox. Install newer version? (Support ended in 2013)<br />
<br />
=== March 4, 2015 ===<br />
'''Updates'''<br />
* [TE] VTK wrapper<br />
** Works on VTK 5.10<br />
** Still need to test VTK 6<br />
** New general wrappers for python functions and classes <br />
<br />
'''Items to Discuss'''<br />
* [RR] Unmark UV-CDAT/VisTrails as a fork of VisTrails/VisTrails?<br />
** When [http://i.imgur.com/UTY2qfg.png filing a pull request] from UV-CDAT/VisTrails, VisTrails/VisTrails is selected by default<br />
** People keep forgetting to change the default ([https://github.com/VisTrails/VisTrails/pull/956 #956] [https://github.com/VisTrails/VisTrails/pull/968 #968] [https://github.com/VisTrails/VisTrails/pull/999 #999] [https://github.com/VisTrails/VisTrails/pull/1000 #1000] [https://github.com/VisTrails/VisTrails/pull/1003 #1003] [https://github.com/VisTrails/VisTrails/pull/1004 #1004] [https://github.com/VisTrails/VisTrails/pull/1005 #1005])<br />
** Only way to change that is to not have it marked as a fork of VisTrails/VisTrails<br />
** Github staff [https://gist.github.com/remram44/b364c24ef1e462ce6e2f can make that change for us]; should we do it?<br />
** Juliana: comments on this? visibility vs. convenience/annoyance for developers<br />
* [RR] What about [https://github.com/VisTrails/VisTrails/issues/991 #991]?<br />
<br />
=== February 25, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] vtkviewcell for infovis support, can we unify with VTKCell?<br />
** need to test this<br />
* [TE] vtk wrapping<br />
** Mostly finished<br />
** VTK 5.10 produce incorrect results with old wrapping<br />
*** Old wrapper is based mostly on VTK 4<br />
*** Most vtk_examples affected<br />
*** [http://www.vtk.org/Wiki/VTK/VTK_6_Migration/Replacement_of_SetInput]<br />
*** should be able to upgrade from SetInput to SetInputData (need to drop GetOutput and replace with self ports)<br />
*** can we change vtkInstance to just return self and not wrap things<br />
** terminator example not working under 5.8?<br />
* How does VTK wrapping fit into general wrapping framework?<br />
* [RR] new persistence package<br />
<br />
=== February 18, 2015 ===<br />
'''Updates'''<br />
* [RR] New VisTrails API and IPython integration ([https://github.com/VisTrails/VisTrails/pull/24 #24])<br />
<br />
'''Items to Discuss'''<br />
* [TE] VTK wrapping<br />
** [https://gist.github.com/rexissimus/16b74693545b5e7c1035 Benchmarking vtk package]<br />
*** Old: 24.7 seconds <br />
*** New: 10.5 seconds (Except first time that adds 8 sec)<br />
**** The parsing that calls is_abstract (that tries to instanciate all vtk classes) is now only run the first time.<br />
**** get_items_from_sigstring takes 2 seconds, maybe we can use a lookup dict for already computed sigstrings?<br />
** Now using a general python function wrapper<br />
*** VTK classes are wrapped into python function that does not depend on vistrails<br />
*** VTK functions can be executed without vistrails<br />
*** The spec maps functions into vistrails modules, but can also describe wrapping <br />
*** A general python function wrapper that supports <br />
**** kwarg inputs<br />
**** single, list, dict outputs<br />
**** callback for progress reporting<br />
**** temporary file generator for using FilePool<br />
**** optional output generation<br />
*** Creating specs:<br />
***** Create spec by hand<br />
***** Auto-create spec outline (TODO) and manually finish it<br />
***** Dynamically create spec (VTK)<br />
***** Implement documentation wrappers (Can use scikitlearn wrapper to wrap numpydoc) (TODO)<br />
***** Classes as bad functions needs to be wrapped in new functions before they are wrapped. This is different for each package.<br />
****** Classes is hard: Like VTK, and matplotlib. Scikit-learn does still not wrap classes<br />
***** Spec diffing and patching could be done using code from matplotlib.<br />
** Still needs upgrades from old VTK package<br />
*** Is it possible to dynamically wrap functions, e.g, you see a SetFunc and just remove the 'Set' prefix. Or do you need to create a complete mapping?<br />
<br />
=== February 11, 2015 ===<br />
'''Updates'''<br />
* Update from Friday's meeting<br />
** discussed VisTrails internals<br />
** discussed wrapping<br />
*** xml discussion, hard to modify because tied to db code<br />
*** TE has made it possible to add the schema-defined attributes to the intermediate representation<br />
*** higher-level operations on the port specs<br />
** make sure the simple case works<br />
*** [JF] take a simple package with documentation and figure out what the base case for wrapping is<br />
<br />
'''Items to Discuss'''<br />
* [TE] VTK wrapping<br />
** Dynamic loading works<br />
** Reading XML is fast enough, but [https://gist.github.com/rexissimus/8272d3c776ccfa115d82 unserializing data is slow]<br />
** Working on patterns for patching<br />
** Matplotlib has many advanced patterns like argument ordering, nested arguments, alternateSpecs, output types.<br />
*** Having all this in a general wrapper might confuse users?<br />
** [RR] Delay module (except for identifiers) until you need it---e.g. don't deal with port specs, etc. until necessary<br />
* Scripting Support [https://github.com/VisTrails/VisTrails/pull/950 #950]<br />
** [RR] Issue with getting code from modules<br />
** Design a simple solution<br />
** [JF] Couldn't you use modules as black boxes without conversion, just to call into modules/subworkflows easily from e.g. IPython?<br />
*** [RR] This is a job for the API, and a very separate use case. See [https://github.com/VisTrails/VisTrails/pull/24 #24]<br />
<br />
=== February 4, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* Wrapping<br />
** Format to use? Currently XML (like current matplotlib)<br />
*** JSON and YAML have simple "to python dictionary" methods<br />
*** But don't stream<br />
*** YAML a lot easier for humans<br />
** [DK] vtk-new-package also changes parameter names, creates enumerations<br />
*** intermediate schema needs to be extensible<br />
*** packages will want to store there specific infos for compute() method generation<br />
*** also might have specs-altering info, like matplotlib's alternateSpec<br />
** representation to code , registry already has schema for some aspects<br />
** [RR] We might want to see if Module subclasses can be created lazily<br />
*** no need to create all the classes just to register them in the registry and never actually use most of them<br />
*** future effort<br />
* [RR] Where should VisTrails packages live?<br />
** tej installs as 'vistrailspkg.tej', TE installed it as 'userpackages.tej'<br />
** Currently, standard packages are 'vistrails.packages.', user packages are 'userpackages.' and packages loaded through pkg_resources might be anything<br />
** [RR] Use 'vistrailspkg.' everywhere?<br />
** Long-term effort to simplify package distribution/installation (and have VisTrails get them automatically?)<br />
<br />
=== January 28, 2015 ===<br />
'''Updates'''<br />
* T. Caswell to come visit on Fri 6 to discuss wrapping work<br />
<br />
'''Items to Discuss'''<br />
* [TE] New VTK wrapping<br />
** Current code by DK seems a good deal faster<br />
** Generates XML that can be patched/tweaked, generates Python code from it, VisTrails only loads generated Python code<br />
** RR would rather have VisTrails load intermediate representation (= XML) directly, wants to make sure this is not slower<br />
** The goal is to turn the intermediate step into something generic that would be used for every wrapped package (vtk, numpy, matplotlib, sklearn, java) instead of each having its own<br />
** TC has its own code at [https://github.com/Nikea/VTTools/blob/master/vttools/wrap_lib.py github:VTTools] which parses numpy docstrings and generates modules, doesn't yet handle classes or persist anything<br />
* Web crawler<br />
** Right now, TE starts jobs for "start crawler", "stop crawler", "install classifier"<br />
** RR would rather have the crawling be a job as far as VisTrails and tej are concerned<br />
** The whole thing would be one pipeline: load examples, train classifier, start crawler [check for job, kill previous one, upload model, start processes], get snapshot, visualize<br />
*** Need some support in tej and job submission system: long-running jobs, stop a job (wait for it to finish?), restart a job even though results are cached<br />
<br />
=== January 21, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Unified wrapping method discussion [https://github.com/VisTrails/VisTrails/issues/991 #991]<br />
** TE to work on reusable method with intermediate representation, starting with VTK<br />
* [RR] Examples for scikit-learn: JF has an old example using Weka with parameter exploration (not currently in source tree)<br />
** AM's examples are enough<br />
* [AM] scikit-learn package is done, merge it in? [https://github.com/VisTrails/VisTrails/pull/955#issuecomment-70286788 #955]<br />
** RR will merge<br />
* [RR] What should copyright headers say? [https://github.com/VisTrails/VisTrails/pull/994#issuecomment-70173935 #994]<br />
** Let's keep everything in there: Utah/Poly/NYU<br />
<br />
=== January 14, 2015 ===<br />
'''Updates'''<br />
* [TE] Working on classifier<br />
* [RR] Scripting integration, work in progress<br />
<br />
'''Items to Discuss'''<br />
* [RR] Unified wrapping method discussion ([https://github.com/VisTrails/VisTrails/issues/991 #991])<br />
** Let's talk next week, [AM] and [DK] are not here<br />
<br />
=== January 7, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* make sure that we address critical issues, questions, and pending review branches in a timely manner<br />
* scripting support<br />
** [RR] no issues if we want to just keep annotations in the generated code to allow the link back to a workflow<br />
** [RR] can translate from workflow to script, working on script to workflow<br />
** will work for parameter value changes, structural changes require changes to the annotations<br />
** need to publish best practices here<br />
** would be cool to do looping in scripts (easier interface than with workflows)<br />
* notebook support (convert form notebook to workflow)<br />
** RR will sync with FC on this<br />
* Issue with console in built-from-scratch<br />
** [TC] iPython rearranged some of the completion stuff in 2.2 and 2.3<br />
** binary has old version of iPython -> 1.0.0, should we update?<br />
* [TC] automated wrapping of numpy and scipy<br />
** discovered a bunch of malformed documentation in numpy and scipy<br />
** has github repo for vistrails tools<br />
** example modules wrap a bunch of R stuff (not baked in, just how things are)<br />
** will be pushing wrapping logic up<br />
** port names forbidden (window and domain)<br />
** have an import hook to get from yaml directly to VisTrails Modules<br />
** should work for any python modules with well-formed numpy docstrings.<br />
* [Action] should make it clear in documentation that Constant now means serializable not that the value doesn't change (e.g. List)<br />
* [TC] might be interesting to try to build components of matplotlib and accumulate in figure (long-term project, but thinking about how this might work)<br />
* [TE] build and build scripts<br />
** completely automatic, buildbot<br />
** need to set the build machines for the environment we want for the binary<br />
** would virtualenv work here?<br />
** [TC] anaconda can pin versions, potential path to test different configurations<br />
** Q: upload nightly binary builds? A: makes sense, make sure they are well-labeled<br />
* sourceforge stats: e.g. http://sourceforge.net/projects/vistrails/files/vistrails/nightly/vistrails-src-nightly.tar.gz/stats/timeline?dates=2014-01-01+to+2015-01-07<br />
* package issues (see Remi's message)<br />
* [TE] Scope of tej<br />
** Support single ssh commands?<br />
** Queue can be used as a remote machine (crawler is using queue.call*)<br />
* SourceForge stats: http://sourceforge.net/projects/vistrails/files/vistrails/nightly/vistrails-src-nightly.tar.gz/stats/timeline?dates=2014-01-01+to+2015-01-07</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=12394Development2016-08-30T21:12:43Z<p>Remi: /* August 31, 2016 */ Websites summary</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2016 ==<br />
=== August 31, 2016 ===<br />
<br />
'''Updates'''<br />
* [RR] Went over vgc.poly.edu/vistrails.org config:<br />
** Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)<br />
** Fixed usersguide scripts (won't disappear again)<br />
** Moved some websites over to HTTPS<br />
** [https://docs.google.com/document/d/18YZewjR2YY9cQ9pzdA-nU9jbu2L7jYcPlDi3iaPadwA/edit Websites]<br />
<br />
'''Items to Discuss'''<br />
* [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/<br />
** Get rid of more services? (svn, trac, ...)<br />
<br />
=== August 17, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Fixed [https://github.com/VisTrails/VisTrails/commit/2cd42d156243067a7663bb0bf0156e5d87188c2f parameter order upgrade bug] that breaks VTK examples.<br />
* [TE] Matplotlib and sklearn are exportable using module-to-script api<br />
* [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.<br />
<br />
'''Items to Discuss'''<br />
* Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!<br />
* Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)<br />
* [TE] Wrap-up notes<br />
<br />
=== August 10, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Python Wrapper and scripting<br />
** All changes merged into python-wrapper branch<br />
*** Tests pass<br />
*** Still need to update sub-branches<br />
** Minor fixes<br />
*** vtkPlot3DReader upgrade bug<br />
*** Added tuple exporting<br />
** Most of vtk examples can be exported [https://gist.github.com/rexissimus/5ac7374cde8525d84f9443f019d234e2 example]<br />
*** Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary<br />
**** Could be a bug in the old vistrail, but not sure<br />
* [DK] Subworkflows and uuids<br />
<br />
=== August 3, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Wrapper patches now stored as [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/vtk_patch.py python code]<br />
** Connection variables now lowercase<br />
** Can we use code as output variables?<br />
*** E.g. "vtk_dataset_mapper.GetOutput(0)"<br />
*** Could mutate output when output method is only supposed to be called once<br />
*** Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)<br />
<br />
=== July 27, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] DB bundle serialization implemented<br />
<br />
'''Items to Discuss'''<br />
* [TE] Finished using patch in script [https://github.com/VisTrails/VisTrails/tree/python-wrapper-script-patches python-wrapper-script-patches]<br />
** locale?<br />
** [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d new patching example] vs [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 old patching example]<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/parse.py#L135-L273 Example of new patches] <br />
** Legacy '.vtkInstance' is now added as a patch "self.vtkInstance=self" (see new patching example)<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/api.py New api for executing standalone modules]<br />
*** Accessing module was disabled for core/api, but a different api for this is probably better anyway.<br />
*** api.Package now gives [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d#file-new_patching-py-L26 directly callable module compute functions]<br />
*** Only works on modules not using interpreter functionality ('.interpreter', etc.)<br />
**** for zip file modules and others that need temp files and use the interpreter for this, need to modify the module, but could work around with python libraries for managing temp files<br />
** Fixed more renaming issues - [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/scripts.py#L63-L64 function argument names should not be replaced]<br />
** Still need to do something nicer for the patch formatting<br />
*** patch keys currently contain non-python-variable characters but this can be fixed<br />
*** Need to replace string template with python variables that is replacable using RedBaron<br />
**** Patches could then look like:<br />
<br />
def vtkInstance():<br />
self.vtkInstance = self<br />
<br />
=== July 20, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] New patch method for python-wrapper [https://gist.github.com/rexissimus/009b303e4cadcd4b8f2fec6cf848cb2b example]<br />
** A patch is a string template with self/input/output variables for patching a class method <br />
** Patches are stored in xml specification<br />
** Patches are assigned to methods in ClassSpec's<br />
** Used both for execution in vistrails and when exporting as script<br />
*** Executed with "exec" when run in vistrails<br />
*** Turns patches into natural calls in script<br />
** Wrapping VTK library no longer needed<br />
*** Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)<br />
** Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)<br />
** Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)<br />
** TODO's<br />
*** Add to export_as_script<br />
*** Patch Modules that represent Python functions as well<br />
<br />
* [vistrails-users] Read data from SQL example?<br />
<br />
=== July 13, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] bundles on sql-alchemy branch<br />
** Bundle, BundleMapping, and BundleSerializer<br />
** New BaseSerializers to deal with versions of directory, zip, db serializers<br />
** Can register the BundleSerializers with the base serializers<br />
** Translation between bundle versions<br />
** Bundle.add_object "just works" if an appropriate mapping was registered<br />
<br />
'''Items to Discuss'''<br />
* [TE] Simplified running module using API for scripting [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 example]<br />
** API Module can be executed using '''compute''' method<br />
*** Takes list of outputs to compute + inputs as keyword arguments<br />
* Status of general cell synchronization (vistrails-users)<br />
<br />
=== July 6, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] Working on bundles<br />
<br />
'''Items to Discuss'''<br />
* [TE] Using vistrails API in scripting [https://gist.github.com/rexissimus/034d9c4dedc3299a079e84981dedd8c7 example 1 (API with VTK)] [https://gist.github.com/rexissimus/a4a9ca6c08d4058c64967d2b35415fd5 example 2 (DownloadFile only)]<br />
** Call module in script using vistrails API<br />
** Can be used when no export method is defined.<br />
* [TE] [https://github.com/VisTrails/VisTrails/commit/c76c109349769fee1280f85c22678380a2f9cd39 Server news bug] Critical?<br />
**[RR] No, unreleased<br />
* [DK] New bundle organization<br />
** BundleMapping contains individual mappings from raw objects to BundleObj<br />
** Must be separate in order to have in-memory, separate from serializer settings<br />
<br />
=== June 29, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as python script<br />
** Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) [https://gist.github.com/rexissimus/16caf5200802e150296f62a0c7844c5a example]<br />
** Preludes (imports) still need improvements<br />
** Works with vtkInteractionHandler (clipping plane)<br />
** Resulting clipping plane is incorrect? (supposed to hide one side of the plane)<br />
** vtkRendererOutput Uses VTKRenderWindow as output<br />
*** Should support different output modes?<br />
*** Does not work with multiple outputs (Next cell is displayed when current is closed)<br />
*** Uses wrapped vtkInstance (as vvtk)<br />
**** Create new patch system with code that can be used both in vistrails and in a python script?<br />
<br />
=== June 22, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fix for upgraded analogies (PR [https://github.com/VisTrails/VisTrails/pull/1177 #1177])<br />
** Uses module location to find module remap (The best we can do right now?)<br />
** Also fixes remapped port names for connections to existing modules <br />
** May still fail on complex upgrades like adding a function on an upgraded module that has a different port name<br />
<br />
* [TE] Working on exporting wrapped libraries (VTK)<br />
<br />
=== June 15, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Added depth logic to python script exporter ([https://github.com/VisTrails/VisTrails/commit/50f220e4f6abe5a7bb8689f73f456f43df2dbc12 full commit])<br />
** Fully implements connection merging and list looping<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Making analogy work on upgraded workflows ([https://github.com/VisTrails/VisTrails/issues/1175 #1175])<br />
** Analogy only works We need to use the non-upgraded action chain<br />
** But pipelines need to be valid (matcher uses module portspecs)<br />
** Tried to get upgrade remap from upgrade action, but this seems impossible<br />
** Use module location instead?<br />
*** Location should be identical for one-to-one module upgrades<br />
<br />
=== June 1, 2016 ===<br />
<br />
'''Updates'''<br />
* VisTrails 2.2.4 announced (finally)<br />
** 5 survey replies<br />
** 47 downloads last week<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as script (python-wrapper)<br />
** to_python_script<br />
*** needs the pipeline (or at least the connections)<br />
**** Should not generate code for any possible VTK method (exporter detects which ports is used in the source)<br />
**** VTK needs to know connection order<br />
**** Adding functions directly to the script results in nicer code<br />
** Add support for multiple input connections<br />
** Add support for list looping<br />
<br />
=== May 25, 2016 ===<br />
<br />
'''Updates'''<br />
* Mailing-list fixed<br />
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])<br />
<br />
'''Items to Discuss'''<br />
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)<br />
* [TE] Union ports<br />
** Show union ports in pipeline (DEMO)<br />
<br />
=== May 18, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] mailing list still down - Julio working on it<br />
* [TE] Improved union port<br />
** New schema 1.0.5 adds portitem.union string attribute<br />
** Works with pipeline view, parameter exploration view, and mashup view<br />
** Should work well for matplotlib<br />
** Use with VTK ports?<br />
*** It uses [item_1,item_2] style ports usually with different tuple length.<br />
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]<br />
<br />
=== May 11, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?<br />
*** Shows single port for all ports with same sort_key<br />
*** Select specific type when adding/showing<br />
*** Only one type should be visible on the module at any time<br />
*** Making tuple parameters union types is probably too complicated<br />
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly) <br />
<br />
* [TE] Feature suggestion: Let connections connect to functions/parameters<br />
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters<br />
** Use separate views for ports and functions (like before)<br />
*** Allow rearranging of functions<br />
** Pros:<br />
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.<br />
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.<br />
**** Makes List module simpler<br />
*** Connect to parameter in tuple<br />
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds. <br />
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.<br />
** Cons:<br />
*** Interface more complicated and crowded<br />
*** More difficult for users to grasp?<br />
*** No immediate use case<br />
*** Makes no union type for tuple parameters a bigger problem.<br />
<br />
* [TE] Publish user survey?<br />
** No response from vistrails release yet<br />
** Not announced on vistrails-users yet.<br />
<br />
=== May 4, 2016 ===<br />
'''Updates'''<br />
* VisTrails 2.2.4 released<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
<br />
=== April 27, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Wrapping Bokeh<br />
** New method type "operation"<br />
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output<br />
**** Glyph sets function name and arguments as output<br />
***** Not serializable, but should always be computed together with the subject of the operation.<br />
**** Figure applies Glyph operation on itself<br />
*** Useful for matplotlib?<br />
**** matplotlib currently find actors by searching upstream?<br />
**** But this is more difficult since matplotlib is not function-based.<br />
** New "union" port type<br />
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value<br />
**** Uses only first value in tuple<br />
**** Bokeh have many properties that support (list, string, Float)<br />
*** Does not work with depth=1 types (e.g. Float of depth 1)<br />
**** Turn Depth 1 types with known type into lists?<br />
<br />
* [TE] New bugfix release<br />
** Finalize survey<br />
<br />
=== April 20, 2016 ===<br />
'''Updates'''<br />
* Merged upgrade fixes<br />
** Now has infinite loop test<br />
** loop test now correctly loops 5 times<br />
*** remap_module used wrong (current) package version when creating ModuleDescriptor<br />
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])<br />
<br />
'''Items to Discuss'''<br />
* New bugfix release<br />
** Merge usagestats<br />
** Need to finalize survey<br />
<br />
=== April 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?<br />
<br />
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])<br />
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]<br />
** Cherry-pick test to preserve source?<br />
** Need maxPipelineFixAttempts option?<br />
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.<br />
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?<br />
*** Would need to be moved into handle_invalid_pipeline<br />
<br />
=== April 6, 2016 ===<br />
'''Updates'''<br />
* [TE] Query View improvements<br />
** Done except for displaying "Version Info" and #1164<br />
* [RR] Working on finishing new interpreter (finally!)<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Try latest upgrade first, then previous.<br />
** If all fail, select latest upgrade?<br />
<br />
=== March 30, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Query View improvements<br />
** Unified do_version_switch and get_upgrades as validate_version()<br />
** Problems with query view using its own controller<br />
*** Fixed module info not being displayed for query results<br />
*** Still no version info for query results<br />
*** Still an issue with version query view not being updated<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== 2015 ==<br />
<br />
=== December 30, 2015 ===<br />
'''Updates'''<br />
* Bokeh package finished<br />
** Can output to Browser/IPython/Spreadsheet (Limited functionality in spreadsheet)<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Bokeh needs QWebEngine (Chromium) in Qt5 which requires PyQt5<br />
*** Many plotting libraries only work in browsers<br />
*** But Chromium may be a big and controversial dependency<br />
*** Or create a browser version of the spreadsheet?<br />
** Some users have requested Python 3<br />
** Not backwards compatible?<br />
** Target for VisTrails 3.0 with new interpreter?<br />
<br />
=== December 16, 2015 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Users list question<br />
* [TE] Wrapping Bokeh<br />
** Added bokeh property parser<br />
*** complements numpydoc parser<br />
*** Not all attributes are available, partially because I am using a pre-release version.<br />
** Wrapped modules<br />
*** bokeh.plotting finished (Backwards like matplotlib: Create plot then draw on it)<br />
*** bokeh.charts mostly done (Missing arguments)<br />
** QWebView mostly works (but no zooming or buttons)<br />
*** It would be better to use external browser or QWebEngineView<br />
** Added improvements to wrapper<br />
*** Generalized port translations (Color/Path to native types)<br />
*** Can use multiple docstring/property parsers simultaneously<br />
<br />
=== December 9, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Looking into wrapper<br />
** Will port TensorFlow<br />
** Doing a pass on the wrapper code & doc<br />
*** UX improvements<br />
** Factorize high-level docstring parsers into core.wrapper? (sphinx, numpy, google)<br />
** Problem with name vs module_name (fails sklearn tests because class_by_name() is broken)<br />
** Looking into compressing specs, lazy registration of modules<br />
<br />
=== December 2, 2015 ===<br />
'''Updates'''<br />
* [TE] Package wrapping<br />
** Wrapped most of numpy/scipy using __all__<br />
** Added class attribute and method access for classes (Can add to class or to separate inspector module)<br />
** TODO: Document wrapping procedure<br />
<br />
'''Items to Discuss'''<br />
<br />
=== November 25, 2015 ===<br />
'''Updates'''<br />
* [RR] New interpreter is coming along<br />
** Everything is a stream<br />
** Task system, eventually work-stealing parallelism<br />
** Streams no longer advance in lock-step<br />
*** Allows for constructions like filter, join, sorted-merge<br />
** depth>1 still to be tested, no current plan to allow this through the module interface<br />
*** Through looping though, can do streams of streams<br />
<br />
'''Items to Discuss'''<br />
* [TE] Package wrapping<br />
** Wrapped most numpy functions (no polynomials)<br />
** and half of scipy classes and functions<br />
** Added functions manually from documentation (no parseable lists?)<br />
** Focusing on data flow constructs (no property/method access for classes)<br />
*** Using spec diff (empty for now) and dynamic parser (stores spec in .vistrails/)<br />
** Python class/function wrapper now stable<br />
** Add google doc parser?<br />
*** Is there a parser available?<br />
<br />
=== November 18, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Package wrapping<br />
** Numpy class wrapping works<br />
*** Basic type is List, since most types are array_likes.<br />
*** TODO: More classes and functions<br />
*** Some operations are in-place and some aren't, and docstring not super clear<br />
*** [RR] It seems only methods mutate, and most have a numpy.xx function equivalent; just manually go through the methods and remove the mutating ones?<br />
** Class wrapper is modular<br />
*** Docstring parser and type string parser can be customized<br />
*** Classes can have optional attribute/method ports<br />
**** Inspectors and attribute/method modules can be created separately<br />
** Use PythonCalc as an example<br />
*** Can be re-implemented as a function with a parseable docstring<br />
*** [RR] PythonCalc is just an example, and it might actually make sense to get rid of it, or do a proper math package with scalar operations (as separate modules, no combobox)<br />
** [RR] TensorFlow to use wrapping as well<br />
*** Very simple (only types 'tensor' & 'variable')<br />
<br />
=== November 11, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Package wrapping<br />
** Wrapping numpy's classes using numpydoc (ndarray)<br />
*** Wrapping constructor arguments, attribute getters/setters as module ports<br />
**** Then how to access attributes afterwards?<br />
***** Use input value port to class modules?<br />
*** Wrap methods as modules? E.g., `ndarray.shape`.<br />
*** Function wrapping is almost a subset of class wrapping<br />
**** May be able to use the same parsing/execution methods<br />
** VTK's non-getter/setter methods could be wrapped as modules<br />
*** We could then remove extra logic in interpreter for keeping function order<br />
** Bokeh uses autogenerated docstrings<br />
*** We may be able to read the specification directly<br />
<br />
* [RR] [https://github.com/VisTrails/VisTrails/pull/1138 TensorFlow package]<br />
** Basic setup working, can execute the Mandelbrot example<br />
** Will autogenerate the operations<br />
<br />
=== November 4, 2015 ===<br />
'''Updates'''<br />
* [RR] Interpreter work requires a fix for [https://github.com/VisTrails/VisTrails/issues/1137 DB issue #1137]<br />
* [DK] Kitware's Resonant [http://www.kitware.com/source/home/post/173]<br />
** [https://github.com/girder/girder Girder]: data management system<br />
** [https://github.com/Kitware/romanesco Romanesco]: execution engine, uses [http://www.celeryproject.org/ Celery] for task management<br />
** Resonant Flow: web app for editing and executing workflows<br />
<br />
'''Items to Discuss'''<br />
* [TE] Added OSX Lion (10.7) VM on build machine using vagrant (seems ok with license?)<br />
** New builds works on Lion<br />
** Change the minimum requirement to 10.7?<br />
* [TE] Package wrapping<br />
* Added upgrade suggestions using 2 spec versions ([http://pastebin.com/30XUU8Gp example])<br />
** using name edit distance to find matches<br />
** Could use something better like port similarity for modules, and type similarity for ports<br />
** Should write python upgrade code?<br />
** Added spec to sklearn package<br />
*** Can now diff spec versions and keep the spec static<br />
*** There will be problems with downgrades and package versions<br />
<br />
=== October 29, 2015 ===<br />
'''Updates'''<br />
* [TE] nightly-build-mac fixed (Needed to approve xcode license)<br />
<br />
'''Items to Discuss'''<br />
* [TE] matplotlib wrapper<br />
** ported to general spec<br />
** Diff can be used on general spec<br />
*** Only needed minor changes<br />
*** Fixed indexing bug that corrupted specs<br />
*** Can now move diff tools to core/wrapper<br />
** Add new plots?<br />
*** New plots in mpl 1.3: eventplot<br />
*** New plots in mpl 1.4: angle_spectrum, magnitude_spectrum, phase_spectrum', violinplot<br />
*** No new plots in mpl 1.5 (from looking at boilerplate.py)<br />
<br />
=== October 21, 2015 ===<br />
'''Updates'''<br />
* [RR] Alexis has arrived, will be working with [RR] on a more efficient interpreter<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1133 New matplotlib package]<br />
** Supporting multiple package versions<br />
*** Check which version can be loaded [https://github.com/VisTrails/VisTrails/pull/1135 #1135]<br />
*** We cannot show version requirements in the list of packages because old packages are loaded when importing the codepath<br />
** Visual diff does not work well, but we can use the spec differ to see differences.<br />
** Unifying vtk and matplotlib wrapping specs (WIP)<br />
*** And porting matplotlibs spec differ to work on the general spec<br />
*** Spec differ can then be used on vtk and other packages in the future <br />
*** Will enable automatic upgrade generation<br />
* [RR] Rework interpreter<br />
** restore abstraction between interpreter/module code<br />
** build looping, streaming, caching into the interpreter<br />
** rework cache<br />
** look into parallelism once basic functionality is there<br />
<br />
=== October 14, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1133 New matplotlib package] (adding numpydoc parser)<br />
<br />
=== October 7, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://github.com/VisTrails/VisTrails/tree/fix-matplotlib-parser New matplotlib parser] (adding numpydoc parser)<br />
** Added numpydoc attribute parser for plots.<br />
*** TODO: Need better port spec reconciliation with call signature parser.<br />
*** We can create a general numpydoc parser, but attribute types are unique to matplotlib.<br />
** Needs package versioning<br />
*** New spec will not support matplotlib < v1.4 due to changed path to axes classes.<br />
*** Load spec version corresponding to installed matplotlib version?<br />
*** Need version downgrades?<br />
<br />
=== September 30, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Updating Matplotlib parser<br />
** Matplotlib docstring parser fails on numpy docstrings<br />
** I have [https://github.com/VisTrails/VisTrails/commit/eae8b0c16260810587e47c41cab53552cbb3cf0a added basic numpydoc parsing]<br />
*** Only used by a few docstrings so far<br />
*** Parsing uses many sources (class tables, signatures, ACCEPTS, method docstrings, definition parsing).<br />
** Caswell said they were thinking about moving to traitlets, but this is not ready yet<br />
<br />
=== September 23, 2015 ===<br />
'''Updates'''<br />
* [TE] VisTrails 2.2.3 released<br />
** Also have pushed to PyPI, binstar, etc.<br />
** Sourceforge vs. GitHub<br />
*** should be able to host releases on GitHub<br />
*** nightly binaries pushed to sf each night<br />
*** old binaries?<br />
* [TE] Re-wrapping MatPlotlib<br />
** Keep static generation - Docstrings are brittle beween matplotlib versions<br />
** Keep generating the executable classes - but create functions, not vistrails Modules <br />
*** Why is there so much patching?<br />
** Re-implemented Alternate PortSpec for InputPortSpec<br />
*** It will now inherit specs from parent (No need to reimplement)<br />
** Module upgrades?<br />
<br />
'''Items to Discuss'''<br />
<br />
=== September 16, 2015 ===<br />
'''Updates'''<br />
* [TE] Job Monitor tests and documentation done<br />
** OK to leave document package in packages directory?<br />
<br />
'''Items to Discuss'''<br />
* [TE] Library wrapping: How to do code patching? [https://docs.google.com/drawings/d/1Sh3eOXot6v2y880jHKQrqXqAwr-0aoqYligg3z8G8MY/edit?usp=sharing Wrapping diagram]<br />
** Insert code into module template (Old VTK code)<br />
** Patch the library that is called (New VTK code)<br />
** Somehow store code in spec and apply when executed?<br />
*** Executed code needs at a minimum access to inputs, outputs, current class<br />
*** Use standard input/output dict and obj reference that the code operates on?<br />
*** Can this be done while keeping the execution abstract?<br />
<br />
* Do new release now?<br />
<br />
=== September 9, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* New release?<br />
** [RR] wants reprounzip<br />
** tej changes & doc<br />
** job monitor & jobmixin fixes<br />
** mongodb<br />
** reprounzip<br />
** warning: MplFigure type is int<br />
** tabledata: add Reshape, DictoToTable & ListToTable, fix ListToTable with numpy arrays<br />
** PythonSource can have same name for an input & output<br />
** don't show spreadsheet at exit<br />
<br />
=== September 2, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Limit autosaves? [https://github.com/VisTrails/VisTrails/pull/1126 #1126]<br />
<br />
=== August 26, 2015 ===<br />
'''Updates'''<br />
* [TE] Added Job support to Parameter Explorations (Requested by Colin), and Mashups<br />
** specify job ids, need to specify different ids for parameter explorations since they have the same version id<br />
** how to deal with parameters passed in on command line<br />
* [General] provenance: should be creating a new version when we execute workflow with changed parameters?<br />
** currently, custom_params annotation stores this in provenance currently<br />
** [RR] API doesn't record provenance if passing in parameters: http://git.io/vsAA5<br />
'''Items to Discuss'''<br />
<br />
=== August 19, 2015 ===<br />
'''Updates'''<br />
* [RR] Improvements to tej, users' guide entry & example [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
* [RR] Internal docs [https://github.com/VisTrails/VisTrails/pull/1116 #1116]<br />
** On ReadTheDocs: [http://vistrails.readthedocs.org/ vistrails.readthedocs.org]<br />
* [TE] Working on Job Monitor documentation<br />
<br />
'''Items to Discuss'''<br />
* [RR] JobMixin and JobMonitor: stable now?<br />
** Definitely needs more tests<br />
* [TE] Stop testing VisTrails 2.0?<br />
<br />
=== August 12, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Internal docs [https://github.com/VisTrails/VisTrails/pull/1116 #1116]<br />
** On ReadTheDocs: [http://vistrails.readthedocs.org/ vistrails.readthedocs.org]<br />
* [RR] JobMixin and JobMonitor: stable now?<br />
** Definitely needs more tests<br />
* [RR] Improvements to tej, users' guide entry & example [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
* [RR] Build broken on Travis; because of IPython 4 released today? ([https://travis-ci.org/VisTrails/VisTrails/builds/75279795 build 992]; [https://github.com/VisTrails/VisTrails/issues/1123 #1123])<br />
<br />
=== August 5, 2015 ===<br />
'''Updates'''<br />
* [TE] Fixes to Jobs<br />
** Could not delete jobs<br />
** Could not run job in group<br />
** Job not reset when calling ModuleSuspended<br />
** Added deleting job from context menu<br />
* [RR] Writing documentation for everything [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
<br />
'''Items to Discuss'''<br />
<br />
=== July 29, 2015 ===<br />
'''Updates'''<br />
* RR still looking into new interpreter thing<br />
** Goal is to take out scheduling logic from Module so it can be split in multiple processes, and so that smarter strategies can be added in time<br />
** This means some work on packages<br />
** Spreadsheet can live in kernel process? Still some UI stuff that will take work (changing configuration, persistent archive's viewer, ...)<br />
* [TE] Problems running examples [https://github.com/VisTrails/VisTrails/pull/1111 #1111]<br />
** Testing of more examples requires additional packages on the test machines.<br />
** Fixed faulty line-ending in PythonSource:s failing on Python 2.6.<br />
** Test suite now testing SUDSWebServices (If web service is down, test suite will fail)<br />
** preferences.py test failed reloading 'dialogs' package, switched to using 'URL'.<br />
<br />
'''Items to Discuss'''<br />
<br />
=== July 22, 2015 ===<br />
'''Updates'''<br />
* [TE] mailing lists back online<br />
* [TE] Working on [https://github.com/VisTrails/VisTrails/issues/1107 #1107]<br />
* [RR] Working on ReproZip package <br />
<br />
'''Items to Discuss'''<br />
* [RR] MongoDB package [https://github.com/VisTrails/VisTrails/pull/1106 #1106]<br />
* [RR] Example for tej docs? [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
<br />
=== July 15, 2015 ===<br />
'''Updates'''<br />
* 2.2.2 released<br />
<br />
'''Items to Discuss'''<br />
* RR is considering executing everything in an IPython kernel (i.e. separate Python interpreter, like the one spawned for a notebook)<br />
** It's a separate process that we can restart without restarting the app/gui<br />
** We can isolate the execution environment (e.g. for the server)<br />
** We can run the whole thing remotely (if your data is elsewhere, just run VisTrails locally and the pipeline elsewhere)<br />
** Pipeline execution no longer makes the interface hang, it just makes the kernel hang (but that's fine)<br />
** We can use notebooks as modules (probably way nicer than the PythonSource module)<br />
** We can run multiple kernels so long as the ports carry things that are serializable<br />
*** meaning we can put the multithreaded-interpreter without all the hacky parts it has now<br />
*** we can run IPython kernels in all the languages IPython supports, [https://github.com/ipython/ipython/wiki/IPython-kernels-for-other-languages currently 46]<br />
<br />
=== July 8, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Buildbot github hook not working after IP address change<br />
* [TE] No reference to VistrailsApplication ([https://github.com/VisTrails/VisTrails/issues/1103 #1103])<br />
** [RR] Added PR [https://github.com/VisTrails/VisTrails/pull/1104 #1104]<br />
* [TE] Reopening VT file after saving with bundled subworkflow won't offer subworkflow upgrade ([https://github.com/VisTrails/VisTrails/issues/1102 #1102])<br />
** allow manual delete to fix right now<br />
** fix this on top of the use-uuid branch<br />
* [TE] Release VisTrails 2.2.1? ([https://github.com/VisTrails/VisTrails/blob/v2.2/CHANGELOG CHANGELOG])<br />
** [https://github.com/VisTrails/VisTrails/pull/1093 output-modules-use-global-dir]<br />
** [https://github.com/VisTrails/VisTrails/milestones/version%202.2 Current issues for 2.2]<br />
<br />
=== July 1, 2015 ===<br />
'''Updates'''<br />
* [TE] PROV fixed<br />
* [TE] Working on subworkflow issues<br />
* [RR] Considering reworking the controller (log vs exception problem, retained upgrades causing interferences) and also the interpreter (IPython?)<br />
<br />
'''Items to Discuss'''<br />
<br />
=== June 24, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* UV-CDAT<br />
* [TE] Can a cyclic workflow be valid? ([https://github.com/VisTrails/VisTrails/issues/1097 #1097])<br />
** focus on disabling the ability to create cyclic pipelines because more things break than just this with a cyclic pipeline<br />
* [RR] Relative paths ([https://github.com/VisTrails/VisTrails/pull/1057 #1057])<br />
** This interacts with the new bundle; how do handle packing files inside the VT bundle?<br />
<br />
=== June 17, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] current_version and reusing existing upgrades are broken ([https://github.com/VisTrails/VisTrails/issues/1095 ticket #1095])<br />
** Could be that export to PROV is using an unflushed upgrade pipeline?<br />
** current_version would then be correct.<br />
** It may work to flush the actions before exporting?<br />
<br />
=== June 10, 2015 ===<br />
'''Updates'''<br />
* DAT: fixed VTK issue on Linux and Mac<br />
* Still crashes on Windows. Need help! Reminder: this works in the VisTrails spreadsheet (QCellPresenter), although no widget get changed there during a drag<br />
** Is it a VTK bug?<br />
** Is it simply impossible to change widgets during the drag, should we do it a different way?<br />
** Did I miss something that is done in VisTrails but somehow not in DAT?<br />
** Low prio, UV-CDAT doesn't run on Windows anyway<br />
<br />
'''Items to Discuss'''<br />
* [TE] Executable Paper ([https://github.com/VisTrails/VisTrails/issues/1086 ticket #1088] [https://github.com/VisTrails/VisTrails/pull/1088 Updated pull request])<br />
* Single instance code and batch mode<br />
<br />
=== June 3, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Executable Paper ([https://github.com/VisTrails/VisTrails/issues/1086 ticket #1088] [https://github.com/VisTrails/VisTrails/pull/1088 pr])<br />
** Requires fixes to command line parameters, Output modules, and batch mode<br />
** How to test this<br />
** Updated missing/outdated flags<br />
** Fixed view issues when generating graphs<br />
* [TE] batch mode<br />
** SpreadsheetOutput not enabled in batch mode. Should we check is_running_gui instead?<br />
** Other instance setting flags from caller<br />
** Is graphsAsPdf replacing spreadsheetDumpPdf?<br />
** graphsAsPdf true by Default?<br />
** Batch mode executing by default (Not needed when generating graphs)<br />
** Re-added workflowInfo as withWorkflowInfo for writing graph and xml workflow<br />
** [DK] batch mode should be outputting to files or stdout, shouldn't always trigger SpreadsheetMode<br />
** execute flag, maybe make execution the default and allow a "--no-execute" if you only want to capture graphs, for example<br />
<br />
=== May 27, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] API question<br />
** http://lists.vistrails.org/pipermail/vistrails-users/2015-May/000357.html<br />
* Status of DAT<br />
** Can add VCS plot to DAT but need to work on configuration windows<br />
** Documentation for graphics templates for UV-CDAT/vcs?<br />
* Qt support?<br />
<br />
=== May 20, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Reviving the DAT, integrating scripting & porting UV-CDAT!<br />
** On [https://github.com/VisTrails/DAT GitHub] ([https://github.com/VisTrails/DAT/issues issues])<br />
** Merging 2 years of development taking longer than expected, but getting there. The plan is to get the patches in VisTrails and never fork again, we never want to get in UV-CDAT's situation (and don't need to).<br />
** VTK cell works fine on Linux but there was flickering on Mac & Windows before; still issues on Mac (Windows status unknown)<br />
** Can get a VCS plot soon (but will need VTK cell fix)<br />
** Integrate in UV-CDAT's build system (so we have cdms, VCS, ...) -> RR can do this, low priority<br />
** How do we integrate scripting?<br />
*** We want to be able to seamlessly make changes to a plot by changing Python code<br />
*** Define new plots by entering Python code without writing modules/packages?<br />
* [RR] UV-CDAT [https://github.com/UV-CDAT/uvcdat/issues/1275 needs] [https://github.com/VisTrails/VisTrails/pull/1073 #1073], please take a look<br />
* [RR] Subworkflow issues<br />
** [https://github.com/VisTrails/VisTrails/issues/1065 #1065]: missing attribute (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/1066 #1066]: groups in subworkflows were ignored when finding dependencies (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/725 #725]: "upgrade subworkflow" button not popping up (fixed); also, namespace issue<br />
** [https://github.com/VisTrails/VisTrails/issues/1071 #1071]: subworkflows don't go through upgrades<br />
* [RR] [https://github.com/VisTrails/VisTrails/issues/1074 #1074]: can't load and edit a single pipeline if upgrade happen<br />
<br />
=== May 13, 2015 ===<br />
'''Updates'''<br />
* [RR] Test skipping whitelist ([https://github.com/VisTrails/VisTrails/pull/1069 #1069]) -- low priority<br />
* [RR] Custom matplotlib modules can't be compatible with both 2.1 and 2.2 ([https://github.com/VisTrails/VisTrails/issues/1067 #1067]); should be fixed for ALPS ([https://github.com/VisTrails/VisTrails/issues/1070 #1070])<br />
<br />
'''Items to Discuss'''<br />
* [RR] UV-CDAT [https://github.com/UV-CDAT/uvcdat/issues/1275 needs] [https://github.com/VisTrails/VisTrails/pull/1073 #1073], please take a look<br />
* [RR] Subworkflow issues<br />
** [https://github.com/VisTrails/VisTrails/issues/1065 #1065]: missing attribute (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/1066 #1066]: groups in subworkflows were ignored when finding dependencies (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/725 #725]: "upgrade subworkflow" button not popping up (fixed); also, namespace issue<br />
** [https://github.com/VisTrails/VisTrails/issues/1071 #1071]: subworkflows don't go through upgrades<br />
* [RR] [https://github.com/VisTrails/VisTrails/issues/1074 #1074]: can't load and edit a single pipeline if upgrade happen<br />
<br />
=== May 7, 2015 ===<br />
'''Updates'''<br />
* [RR] Export/import workflow to Python working!<br />
<br />
'''Items to Discuss'''<br />
* [Claudio] UV-CDAT<br />
** The UV-CDAT project is the biggest user base of VisTrails<br />
** VisTrails package management provides a lot of friction towards people plugging in their code<br />
** Need to make it easy to integrate your random Python scripts in the system without having to deal with all the boilerplate, at least in the first step<br />
** [RR] argues that modules are still good; UV-CDAT shouldn't move towards a purely script-based backend<br />
** [RR] export/import with Python could reduce a lot of that friction by allowing 1) to edit workflow as Python 2) to open up boxes automatically if needed code doesn't match actual modules<br />
** ...<br />
<br />
=== April 29, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] <strike>Abstractions</strike> subworkflows status ([https://github.com/VisTrails/VisTrails/labels/subworkflows tickets])<br />
* [RR] matplotlib compatibility (2.1 & 2.2), [https://github.com/VisTrails/VisTrails/issues/1067 #1067]<br />
** RR to try and fix ALPS matplotlib modules<br />
<br />
=== April 22, 2015 ===<br />
'''Updates'''<br />
* 2.2.0 has been released!<br />
* Windows issue (via email from Ryan)<br />
** issue with manifest file (may be a new file in VTK6?)<br />
** Tommy has regenerated new Windows builds<br />
* Binaries, pypi, and conda released<br />
* [RR] Export as script<br />
* Python sources using VTK need to switch to SetInputData (users should be aware of this)<br />
<br />
=== April 15, 2015 ===<br />
'''Updates'''<br />
* Ready for 2.2.0 (apart from binary/deps issues)<br />
** Missing some libs (scikit-learn, tej, tdparser, SQLAlchemy+connectors<br />
** Windows: runvistrails.py is no longer used, so the PATH is wrong<br />
** Windows: pip is broken, but it probably wouldn't work anyway because of permission issues (disable this?)<br />
* Queries, upgrades and getPipeline() usage ([https://github.com/VisTrails/VisTrails/issues/1054 #1054])<br />
** Getting a pipeline with getPipeline() is not safe: it might return an invalid pipeline<br />
** This is used in many places throughout the code, like queries<br />
** Upgrading would require going through the controller, but that creates new actions<br />
* [TC] Avoid copying a module's output if it's used as input by exactly one downstream module ([https://github.com/VisTrails/VisTrails/issues/1060 #1060]) (useful for big numpy arrays you can update in-place)<br />
<br />
=== April 8, 2015 ===<br />
'''Updates'''<br />
* [DK] Merged RR's changes for output modules (1012 and 1013)<br />
** RR will merge remaining changes, then create v2.2 branch!<br />
<br />
'''Items to Discuss'''<br />
* BNL need numpy array to VTK image<br />
** Looks like VTK has some helpers for this<br />
** We will help if issues arise<br />
** Will contribute back to VisTrails package<br />
* Upgrade issue: [https://github.com/VisTrails/VisTrails/issues/1017 #1017]<br />
** Automatic upgrades should happen between versions of provided upgrades<br />
** Our existing upgrades work around this so it doesn't need to be 2.2.0<br />
* Corner-case VTK modules<br />
** No longer need VTKCell input port, so don't interfere with registry and API anymore<br />
** Still work weirdly, people probably shouldn't use them<br />
** But we have lots of clunky modules since we wrap the whole of VTK; some people might rely on this and know how to use them, let's keep them anyway<br />
** Ready for 2.2.0<br />
<br />
=== April 1, 2015 ===<br />
'''Updates'''<br />
* [RR] UV-CDAT: bugfixing for 2.2, long-term plans: implement scripting import/export in VisTrails, port to UV-CDAT<br />
* Possibly, try to move to regular VisTrails to use new features<br />
<br />
'''Items to Discuss'''<br />
* 2.2 release: nothing much is pending anymore, release next week?<br />
** [TE] vtkExporter classes [https://github.com/VisTrails/VisTrails/issues/1032 #1032]<br />
** [RR] Hiding upgrades in version tree might make it (or might be 2.2.1 so we can test it out) [https://github.com/VisTrails/VisTrails/issues/949 #949]<br />
** Output modules changes to go in<br />
* add note to documentation about order of parameters in VTK<br />
* add issue about exporter upgrades if not already there<br />
<br />
=== March 25, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* 2.2 release<br />
** Checklist on Github: https://github.com/VisTrails/VisTrails/wiki/2.2-Release-Checklist<br />
** Review output modules<br />
** RR has a couple more issues to fix<br />
** Ready to go -- sign app package for OSX? [https://github.com/VisTrails/VisTrails/issues/984 #984] We need access to the Apple Membership team<br />
<br />
=== March 18, 2015 ===<br />
'''Updates'''<br />
* [TE] VTK6 works<br />
<br />
'''Items to Discuss'''<br />
* [RR] Release v2.1.5 with backported tabledata?<br />
** MTA example needs updated tabledata (for JoinTables)<br />
** Google Maps package still not available<br />
* [RR] Work torward v2.2.0?<br />
** changes:<br />
*** new persistence<br />
*** API changes<br />
*** output module changes (upgrades?), maintain cells but try to upgrade<br />
*** not wrapping stuff<br />
*** VTK6? yes<br />
*** JobSubmission stuff?<br />
*** relabeling for upgrades [https://github.com/VisTrails/VisTrails/issues/949 #949]<br />
** makes sense, needs the tree view code to be updated, check selection<br />
** See [https://github.com/VisTrails/VisTrails/wiki/2.2-Release-Checklist 2.2 checklist]<br />
* Discussion of [https://github.com/VisTrails/VisTrails/issues/1016 #1016]<br />
** plumbing between outputs and output modes, how to define a mode that works for many outputs without writing for each output?<br />
<br />
=== March 11, 2015 ===<br />
'''Updates'''<br />
* [RR] persistent_archive done; merge? ([https://github.com/VisTrails/VisTrails/pull/755 #755])<br />
** note about the focus events for widgets<br />
** TE be aware of file_archive for future binaries that include persistent_archive<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/998 New VTK package] finished<br />
<br />
'''Items to Discuss'''<br />
* [RR] Unmark UV-CDAT/VisTrails as a fork of VisTrails/VisTrails? (see last week; decision needed)<br />
** RR email to JF about this<br />
** yes; email sent to Github<br />
* [RR] Switching order of output ports ([https://github.com/VisTrails/VisTrails/pull/1006 #1006])<br />
** added port specs are sorted at a separate spot (Module.*_port_specs properties) than those in the registry (which are sorted in the registry), but those two lists are just combined without respect to sort keys<br />
** need to determine whether the two lists should be merged or remain distinct<br />
** should make sure that order of input ports and output ports makes parallel connections for things with same order<br />
** DK suggests breaking backward compatibility here: workflows still run, can fix easily if a problem in an existing package.<br />
* [RR] Question about output modes ([https://github.com/VisTrails/VisTrails/pull/1007 #1007]), how to integrate in API ([https://github.com/VisTrails/VisTrails/pull/24 #24])<br />
** Should ImageFileMode be removed? ("image" is not a mode, "file" is)<br />
** ImageOutput missing?<br />
** Feel free to change how formats works<br />
* [TE] Test suite segfaults on Fedora 17 virtualbox. Install newer version? (Support ended in 2013)<br />
<br />
=== March 4, 2015 ===<br />
'''Updates'''<br />
* [TE] VTK wrapper<br />
** Works on VTK 5.10<br />
** Still need to test VTK 6<br />
** New general wrappers for python functions and classes <br />
<br />
'''Items to Discuss'''<br />
* [RR] Unmark UV-CDAT/VisTrails as a fork of VisTrails/VisTrails?<br />
** When [http://i.imgur.com/UTY2qfg.png filing a pull request] from UV-CDAT/VisTrails, VisTrails/VisTrails is selected by default<br />
** People keep forgetting to change the default ([https://github.com/VisTrails/VisTrails/pull/956 #956] [https://github.com/VisTrails/VisTrails/pull/968 #968] [https://github.com/VisTrails/VisTrails/pull/999 #999] [https://github.com/VisTrails/VisTrails/pull/1000 #1000] [https://github.com/VisTrails/VisTrails/pull/1003 #1003] [https://github.com/VisTrails/VisTrails/pull/1004 #1004] [https://github.com/VisTrails/VisTrails/pull/1005 #1005])<br />
** Only way to change that is to not have it marked as a fork of VisTrails/VisTrails<br />
** Github staff [https://gist.github.com/remram44/b364c24ef1e462ce6e2f can make that change for us]; should we do it?<br />
** Juliana: comments on this? visibility vs. convenience/annoyance for developers<br />
* [RR] What about [https://github.com/VisTrails/VisTrails/issues/991 #991]?<br />
<br />
=== February 25, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] vtkviewcell for infovis support, can we unify with VTKCell?<br />
** need to test this<br />
* [TE] vtk wrapping<br />
** Mostly finished<br />
** VTK 5.10 produce incorrect results with old wrapping<br />
*** Old wrapper is based mostly on VTK 4<br />
*** Most vtk_examples affected<br />
*** [http://www.vtk.org/Wiki/VTK/VTK_6_Migration/Replacement_of_SetInput]<br />
*** should be able to upgrade from SetInput to SetInputData (need to drop GetOutput and replace with self ports)<br />
*** can we change vtkInstance to just return self and not wrap things<br />
** terminator example not working under 5.8?<br />
* How does VTK wrapping fit into general wrapping framework?<br />
* [RR] new persistence package<br />
<br />
=== February 18, 2015 ===<br />
'''Updates'''<br />
* [RR] New VisTrails API and IPython integration ([https://github.com/VisTrails/VisTrails/pull/24 #24])<br />
<br />
'''Items to Discuss'''<br />
* [TE] VTK wrapping<br />
** [https://gist.github.com/rexissimus/16b74693545b5e7c1035 Benchmarking vtk package]<br />
*** Old: 24.7 seconds <br />
*** New: 10.5 seconds (Except first time that adds 8 sec)<br />
**** The parsing that calls is_abstract (that tries to instanciate all vtk classes) is now only run the first time.<br />
**** get_items_from_sigstring takes 2 seconds, maybe we can use a lookup dict for already computed sigstrings?<br />
** Now using a general python function wrapper<br />
*** VTK classes are wrapped into python function that does not depend on vistrails<br />
*** VTK functions can be executed without vistrails<br />
*** The spec maps functions into vistrails modules, but can also describe wrapping <br />
*** A general python function wrapper that supports <br />
**** kwarg inputs<br />
**** single, list, dict outputs<br />
**** callback for progress reporting<br />
**** temporary file generator for using FilePool<br />
**** optional output generation<br />
*** Creating specs:<br />
***** Create spec by hand<br />
***** Auto-create spec outline (TODO) and manually finish it<br />
***** Dynamically create spec (VTK)<br />
***** Implement documentation wrappers (Can use scikitlearn wrapper to wrap numpydoc) (TODO)<br />
***** Classes as bad functions needs to be wrapped in new functions before they are wrapped. This is different for each package.<br />
****** Classes is hard: Like VTK, and matplotlib. Scikit-learn does still not wrap classes<br />
***** Spec diffing and patching could be done using code from matplotlib.<br />
** Still needs upgrades from old VTK package<br />
*** Is it possible to dynamically wrap functions, e.g, you see a SetFunc and just remove the 'Set' prefix. Or do you need to create a complete mapping?<br />
<br />
=== February 11, 2015 ===<br />
'''Updates'''<br />
* Update from Friday's meeting<br />
** discussed VisTrails internals<br />
** discussed wrapping<br />
*** xml discussion, hard to modify because tied to db code<br />
*** TE has made it possible to add the schema-defined attributes to the intermediate representation<br />
*** higher-level operations on the port specs<br />
** make sure the simple case works<br />
*** [JF] take a simple package with documentation and figure out what the base case for wrapping is<br />
<br />
'''Items to Discuss'''<br />
* [TE] VTK wrapping<br />
** Dynamic loading works<br />
** Reading XML is fast enough, but [https://gist.github.com/rexissimus/8272d3c776ccfa115d82 unserializing data is slow]<br />
** Working on patterns for patching<br />
** Matplotlib has many advanced patterns like argument ordering, nested arguments, alternateSpecs, output types.<br />
*** Having all this in a general wrapper might confuse users?<br />
** [RR] Delay module (except for identifiers) until you need it---e.g. don't deal with port specs, etc. until necessary<br />
* Scripting Support [https://github.com/VisTrails/VisTrails/pull/950 #950]<br />
** [RR] Issue with getting code from modules<br />
** Design a simple solution<br />
** [JF] Couldn't you use modules as black boxes without conversion, just to call into modules/subworkflows easily from e.g. IPython?<br />
*** [RR] This is a job for the API, and a very separate use case. See [https://github.com/VisTrails/VisTrails/pull/24 #24]<br />
<br />
=== February 4, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* Wrapping<br />
** Format to use? Currently XML (like current matplotlib)<br />
*** JSON and YAML have simple "to python dictionary" methods<br />
*** But don't stream<br />
*** YAML a lot easier for humans<br />
** [DK] vtk-new-package also changes parameter names, creates enumerations<br />
*** intermediate schema needs to be extensible<br />
*** packages will want to store there specific infos for compute() method generation<br />
*** also might have specs-altering info, like matplotlib's alternateSpec<br />
** representation to code , registry already has schema for some aspects<br />
** [RR] We might want to see if Module subclasses can be created lazily<br />
*** no need to create all the classes just to register them in the registry and never actually use most of them<br />
*** future effort<br />
* [RR] Where should VisTrails packages live?<br />
** tej installs as 'vistrailspkg.tej', TE installed it as 'userpackages.tej'<br />
** Currently, standard packages are 'vistrails.packages.', user packages are 'userpackages.' and packages loaded through pkg_resources might be anything<br />
** [RR] Use 'vistrailspkg.' everywhere?<br />
** Long-term effort to simplify package distribution/installation (and have VisTrails get them automatically?)<br />
<br />
=== January 28, 2015 ===<br />
'''Updates'''<br />
* T. Caswell to come visit on Fri 6 to discuss wrapping work<br />
<br />
'''Items to Discuss'''<br />
* [TE] New VTK wrapping<br />
** Current code by DK seems a good deal faster<br />
** Generates XML that can be patched/tweaked, generates Python code from it, VisTrails only loads generated Python code<br />
** RR would rather have VisTrails load intermediate representation (= XML) directly, wants to make sure this is not slower<br />
** The goal is to turn the intermediate step into something generic that would be used for every wrapped package (vtk, numpy, matplotlib, sklearn, java) instead of each having its own<br />
** TC has its own code at [https://github.com/Nikea/VTTools/blob/master/vttools/wrap_lib.py github:VTTools] which parses numpy docstrings and generates modules, doesn't yet handle classes or persist anything<br />
* Web crawler<br />
** Right now, TE starts jobs for "start crawler", "stop crawler", "install classifier"<br />
** RR would rather have the crawling be a job as far as VisTrails and tej are concerned<br />
** The whole thing would be one pipeline: load examples, train classifier, start crawler [check for job, kill previous one, upload model, start processes], get snapshot, visualize<br />
*** Need some support in tej and job submission system: long-running jobs, stop a job (wait for it to finish?), restart a job even though results are cached<br />
<br />
=== January 21, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Unified wrapping method discussion [https://github.com/VisTrails/VisTrails/issues/991 #991]<br />
** TE to work on reusable method with intermediate representation, starting with VTK<br />
* [RR] Examples for scikit-learn: JF has an old example using Weka with parameter exploration (not currently in source tree)<br />
** AM's examples are enough<br />
* [AM] scikit-learn package is done, merge it in? [https://github.com/VisTrails/VisTrails/pull/955#issuecomment-70286788 #955]<br />
** RR will merge<br />
* [RR] What should copyright headers say? [https://github.com/VisTrails/VisTrails/pull/994#issuecomment-70173935 #994]<br />
** Let's keep everything in there: Utah/Poly/NYU<br />
<br />
=== January 14, 2015 ===<br />
'''Updates'''<br />
* [TE] Working on classifier<br />
* [RR] Scripting integration, work in progress<br />
<br />
'''Items to Discuss'''<br />
* [RR] Unified wrapping method discussion ([https://github.com/VisTrails/VisTrails/issues/991 #991])<br />
** Let's talk next week, [AM] and [DK] are not here<br />
<br />
=== January 7, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* make sure that we address critical issues, questions, and pending review branches in a timely manner<br />
* scripting support<br />
** [RR] no issues if we want to just keep annotations in the generated code to allow the link back to a workflow<br />
** [RR] can translate from workflow to script, working on script to workflow<br />
** will work for parameter value changes, structural changes require changes to the annotations<br />
** need to publish best practices here<br />
** would be cool to do looping in scripts (easier interface than with workflows)<br />
* notebook support (convert form notebook to workflow)<br />
** RR will sync with FC on this<br />
* Issue with console in built-from-scratch<br />
** [TC] iPython rearranged some of the completion stuff in 2.2 and 2.3<br />
** binary has old version of iPython -> 1.0.0, should we update?<br />
* [TC] automated wrapping of numpy and scipy<br />
** discovered a bunch of malformed documentation in numpy and scipy<br />
** has github repo for vistrails tools<br />
** example modules wrap a bunch of R stuff (not baked in, just how things are)<br />
** will be pushing wrapping logic up<br />
** port names forbidden (window and domain)<br />
** have an import hook to get from yaml directly to VisTrails Modules<br />
** should work for any python modules with well-formed numpy docstrings.<br />
* [Action] should make it clear in documentation that Constant now means serializable not that the value doesn't change (e.g. List)<br />
* [TC] might be interesting to try to build components of matplotlib and accumulate in figure (long-term project, but thinking about how this might work)<br />
* [TE] build and build scripts<br />
** completely automatic, buildbot<br />
** need to set the build machines for the environment we want for the binary<br />
** would virtualenv work here?<br />
** [TC] anaconda can pin versions, potential path to test different configurations<br />
** Q: upload nightly binary builds? A: makes sense, make sure they are well-labeled<br />
* sourceforge stats: e.g. http://sourceforge.net/projects/vistrails/files/vistrails/nightly/vistrails-src-nightly.tar.gz/stats/timeline?dates=2014-01-01+to+2015-01-07<br />
* package issues (see Remi's message)<br />
* [TE] Scope of tej<br />
** Support single ssh commands?<br />
** Queue can be used as a remote machine (crawler is using queue.call*)<br />
* SourceForge stats: http://sourceforge.net/projects/vistrails/files/vistrails/nightly/vistrails-src-nightly.tar.gz/stats/timeline?dates=2014-01-01+to+2015-01-07<br />
<br />
== Older meetings ==<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=12200Development2016-07-27T15:09:50Z<p>Remi: /* July 27, 2016 */</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2016 ==<br />
<br />
=== July 27, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] DB bundle serialization implemented<br />
<br />
'''Items to Discuss'''<br />
* [TE] Finished using patch in script [https://github.com/VisTrails/VisTrails/tree/python-wrapper-script-patches python-wrapper-script-patches]<br />
** locale?<br />
** [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d new patching example] vs [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 old patching example]<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/parse.py#L135-L273 Example of new patches] <br />
** Legacy '.vtkInstance' is now added as a patch "self.vtkInstance=self" (see new patching example)<br />
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/api.py New api for executing standalone modules]<br />
*** Accessing module was disabled for core/api, but a different api for this is probably better anyway.<br />
*** api.Package now gives [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d#file-new_patching-py-L26 directly callable module compute functions]<br />
*** Only works on modules not using interpreter functionality ('.interpreter', etc.)<br />
** Fixed more renaming issues - [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/scripts.py#L63-L64 function argument names should not be replaced]<br />
** Still need to do something nicer for the patch formatting<br />
*** patch keys currently contain non-python-variable characters but this can be fixed<br />
*** Need to replace string template with python variables that is replacable using RedBaron<br />
**** Patches could then look like:<br />
<br />
def vtkInstance():<br />
self.vtkInstance = self<br />
<br />
=== July 20, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] New patch method for python-wrapper [https://gist.github.com/rexissimus/009b303e4cadcd4b8f2fec6cf848cb2b example]<br />
** A patch is a string template with self/input/output variables for patching a class method <br />
** Patches are stored in xml specification<br />
** Patches are assigned to methods in ClassSpec's<br />
** Used both for execution in vistrails and when exporting as script<br />
*** Executed with "exec" when run in vistrails<br />
*** Turns patches into natural calls in script<br />
** Wrapping VTK library no longer needed<br />
*** Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)<br />
** Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)<br />
** Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)<br />
** TODO's<br />
*** Add to export_as_script<br />
*** Patch Modules that represent Python functions as well<br />
<br />
* [vistrails-users] Read data from SQL example?<br />
<br />
=== July 13, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] bundles on sql-alchemy branch<br />
** Bundle, BundleMapping, and BundleSerializer<br />
** New BaseSerializers to deal with versions of directory, zip, db serializers<br />
** Can register the BundleSerializers with the base serializers<br />
** Translation between bundle versions<br />
** Bundle.add_object "just works" if an appropriate mapping was registered<br />
<br />
'''Items to Discuss'''<br />
* [TE] Simplified running module using API for scripting [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 example]<br />
** API Module can be executed using '''compute''' method<br />
*** Takes list of outputs to compute + inputs as keyword arguments<br />
* Status of general cell synchronization (vistrails-users)<br />
<br />
=== July 6, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] Working on bundles<br />
<br />
'''Items to Discuss'''<br />
* [TE] Using vistrails API in scripting [https://gist.github.com/rexissimus/034d9c4dedc3299a079e84981dedd8c7 example 1 (API with VTK)] [https://gist.github.com/rexissimus/a4a9ca6c08d4058c64967d2b35415fd5 example 2 (DownloadFile only)]<br />
** Call module in script using vistrails API<br />
** Can be used when no export method is defined.<br />
* [TE] [https://github.com/VisTrails/VisTrails/commit/c76c109349769fee1280f85c22678380a2f9cd39 Server news bug] Critical?<br />
**[RR] No, unreleased<br />
* [DK] New bundle organization<br />
** BundleMapping contains individual mappings from raw objects to BundleObj<br />
** Must be separate in order to have in-memory, separate from serializer settings<br />
<br />
=== June 29, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as python script<br />
** Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) [https://gist.github.com/rexissimus/16caf5200802e150296f62a0c7844c5a example]<br />
** Preludes (imports) still need improvements<br />
** Works with vtkInteractionHandler (clipping plane)<br />
** Resulting clipping plane is incorrect? (supposed to hide one side of the plane)<br />
** vtkRendererOutput Uses VTKRenderWindow as output<br />
*** Should support different output modes?<br />
*** Does not work with multiple outputs (Next cell is displayed when current is closed)<br />
*** Uses wrapped vtkInstance (as vvtk)<br />
**** Create new patch system with code that can be used both in vistrails and in a python script?<br />
<br />
=== June 22, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fix for upgraded analogies (PR [https://github.com/VisTrails/VisTrails/pull/1177 #1177])<br />
** Uses module location to find module remap (The best we can do right now?)<br />
** Also fixes remapped port names for connections to existing modules <br />
** May still fail on complex upgrades like adding a function on an upgraded module that has a different port name<br />
<br />
* [TE] Working on exporting wrapped libraries (VTK)<br />
<br />
=== June 15, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Added depth logic to python script exporter ([https://github.com/VisTrails/VisTrails/commit/50f220e4f6abe5a7bb8689f73f456f43df2dbc12 full commit])<br />
** Fully implements connection merging and list looping<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Making analogy work on upgraded workflows ([https://github.com/VisTrails/VisTrails/issues/1175 #1175])<br />
** Analogy only works We need to use the non-upgraded action chain<br />
** But pipelines need to be valid (matcher uses module portspecs)<br />
** Tried to get upgrade remap from upgrade action, but this seems impossible<br />
** Use module location instead?<br />
*** Location should be identical for one-to-one module upgrades<br />
<br />
=== June 1, 2016 ===<br />
<br />
'''Updates'''<br />
* VisTrails 2.2.4 announced (finally)<br />
** 5 survey replies<br />
** 47 downloads last week<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as script (python-wrapper)<br />
** to_python_script<br />
*** needs the pipeline (or at least the connections)<br />
**** Should not generate code for any possible VTK method (exporter detects which ports is used in the source)<br />
**** VTK needs to know connection order<br />
**** Adding functions directly to the script results in nicer code<br />
** Add support for multiple input connections<br />
** Add support for list looping<br />
<br />
=== May 25, 2016 ===<br />
<br />
'''Updates'''<br />
* Mailing-list fixed<br />
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])<br />
<br />
'''Items to Discuss'''<br />
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)<br />
* [TE] Union ports<br />
** Show union ports in pipeline (DEMO)<br />
<br />
=== May 18, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] mailing list still down - Julio working on it<br />
* [TE] Improved union port<br />
** New schema 1.0.5 adds portitem.union string attribute<br />
** Works with pipeline view, parameter exploration view, and mashup view<br />
** Should work well for matplotlib<br />
** Use with VTK ports?<br />
*** It uses [item_1,item_2] style ports usually with different tuple length.<br />
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]<br />
<br />
=== May 11, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?<br />
*** Shows single port for all ports with same sort_key<br />
*** Select specific type when adding/showing<br />
*** Only one type should be visible on the module at any time<br />
*** Making tuple parameters union types is probably too complicated<br />
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly) <br />
<br />
* [TE] Feature suggestion: Let connections connect to functions/parameters<br />
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters<br />
** Use separate views for ports and functions (like before)<br />
*** Allow rearranging of functions<br />
** Pros:<br />
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.<br />
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.<br />
**** Makes List module simpler<br />
*** Connect to parameter in tuple<br />
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds. <br />
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.<br />
** Cons:<br />
*** Interface more complicated and crowded<br />
*** More difficult for users to grasp?<br />
*** No immediate use case<br />
*** Makes no union type for tuple parameters a bigger problem.<br />
<br />
* [TE] Publish user survey?<br />
** No response from vistrails release yet<br />
** Not announced on vistrails-users yet.<br />
<br />
=== May 4, 2016 ===<br />
'''Updates'''<br />
* VisTrails 2.2.4 released<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
<br />
=== April 27, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Wrapping Bokeh<br />
** New method type "operation"<br />
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output<br />
**** Glyph sets function name and arguments as output<br />
***** Not serializable, but should always be computed together with the subject of the operation.<br />
**** Figure applies Glyph operation on itself<br />
*** Useful for matplotlib?<br />
**** matplotlib currently find actors by searching upstream?<br />
**** But this is more difficult since matplotlib is not function-based.<br />
** New "union" port type<br />
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value<br />
**** Uses only first value in tuple<br />
**** Bokeh have many properties that support (list, string, Float)<br />
*** Does not work with depth=1 types (e.g. Float of depth 1)<br />
**** Turn Depth 1 types with known type into lists?<br />
<br />
* [TE] New bugfix release<br />
** Finalize survey<br />
<br />
=== April 20, 2016 ===<br />
'''Updates'''<br />
* Merged upgrade fixes<br />
** Now has infinite loop test<br />
** loop test now correctly loops 5 times<br />
*** remap_module used wrong (current) package version when creating ModuleDescriptor<br />
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])<br />
<br />
'''Items to Discuss'''<br />
* New bugfix release<br />
** Merge usagestats<br />
** Need to finalize survey<br />
<br />
=== April 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?<br />
<br />
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])<br />
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]<br />
** Cherry-pick test to preserve source?<br />
** Need maxPipelineFixAttempts option?<br />
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.<br />
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?<br />
*** Would need to be moved into handle_invalid_pipeline<br />
<br />
=== April 6, 2016 ===<br />
'''Updates'''<br />
* [TE] Query View improvements<br />
** Done except for displaying "Version Info" and #1164<br />
* [RR] Working on finishing new interpreter (finally!)<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Try latest upgrade first, then previous.<br />
** If all fail, select latest upgrade?<br />
<br />
=== March 30, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Query View improvements<br />
** Unified do_version_switch and get_upgrades as validate_version()<br />
** Problems with query view using its own controller<br />
*** Fixed module info not being displayed for query results<br />
*** Still no version info for query results<br />
*** Still an issue with version query view not being updated<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== 2015 ==<br />
<br />
=== December 30, 2015 ===<br />
'''Updates'''<br />
* Bokeh package finished<br />
** Can output to Browser/IPython/Spreadsheet (Limited functionality in spreadsheet)<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Bokeh needs QWebEngine (Chromium) in Qt5 which requires PyQt5<br />
*** Many plotting libraries only work in browsers<br />
*** But Chromium may be a big and controversial dependency<br />
*** Or create a browser version of the spreadsheet?<br />
** Some users have requested Python 3<br />
** Not backwards compatible?<br />
** Target for VisTrails 3.0 with new interpreter?<br />
<br />
=== December 16, 2015 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Users list question<br />
* [TE] Wrapping Bokeh<br />
** Added bokeh property parser<br />
*** complements numpydoc parser<br />
*** Not all attributes are available, partially because I am using a pre-release version.<br />
** Wrapped modules<br />
*** bokeh.plotting finished (Backwards like matplotlib: Create plot then draw on it)<br />
*** bokeh.charts mostly done (Missing arguments)<br />
** QWebView mostly works (but no zooming or buttons)<br />
*** It would be better to use external browser or QWebEngineView<br />
** Added improvements to wrapper<br />
*** Generalized port translations (Color/Path to native types)<br />
*** Can use multiple docstring/property parsers simultaneously<br />
<br />
=== December 9, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Looking into wrapper<br />
** Will port TensorFlow<br />
** Doing a pass on the wrapper code & doc<br />
*** UX improvements<br />
** Factorize high-level docstring parsers into core.wrapper? (sphinx, numpy, google)<br />
** Problem with name vs module_name (fails sklearn tests because class_by_name() is broken)<br />
** Looking into compressing specs, lazy registration of modules<br />
<br />
=== December 2, 2015 ===<br />
'''Updates'''<br />
* [TE] Package wrapping<br />
** Wrapped most of numpy/scipy using __all__<br />
** Added class attribute and method access for classes (Can add to class or to separate inspector module)<br />
** TODO: Document wrapping procedure<br />
<br />
'''Items to Discuss'''<br />
<br />
=== November 25, 2015 ===<br />
'''Updates'''<br />
* [RR] New interpreter is coming along<br />
** Everything is a stream<br />
** Task system, eventually work-stealing parallelism<br />
** Streams no longer advance in lock-step<br />
*** Allows for constructions like filter, join, sorted-merge<br />
** depth>1 still to be tested, no current plan to allow this through the module interface<br />
*** Through looping though, can do streams of streams<br />
<br />
'''Items to Discuss'''<br />
* [TE] Package wrapping<br />
** Wrapped most numpy functions (no polynomials)<br />
** and half of scipy classes and functions<br />
** Added functions manually from documentation (no parseable lists?)<br />
** Focusing on data flow constructs (no property/method access for classes)<br />
*** Using spec diff (empty for now) and dynamic parser (stores spec in .vistrails/)<br />
** Python class/function wrapper now stable<br />
** Add google doc parser?<br />
*** Is there a parser available?<br />
<br />
=== November 18, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Package wrapping<br />
** Numpy class wrapping works<br />
*** Basic type is List, since most types are array_likes.<br />
*** TODO: More classes and functions<br />
*** Some operations are in-place and some aren't, and docstring not super clear<br />
*** [RR] It seems only methods mutate, and most have a numpy.xx function equivalent; just manually go through the methods and remove the mutating ones?<br />
** Class wrapper is modular<br />
*** Docstring parser and type string parser can be customized<br />
*** Classes can have optional attribute/method ports<br />
**** Inspectors and attribute/method modules can be created separately<br />
** Use PythonCalc as an example<br />
*** Can be re-implemented as a function with a parseable docstring<br />
*** [RR] PythonCalc is just an example, and it might actually make sense to get rid of it, or do a proper math package with scalar operations (as separate modules, no combobox)<br />
** [RR] TensorFlow to use wrapping as well<br />
*** Very simple (only types 'tensor' & 'variable')<br />
<br />
=== November 11, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Package wrapping<br />
** Wrapping numpy's classes using numpydoc (ndarray)<br />
*** Wrapping constructor arguments, attribute getters/setters as module ports<br />
**** Then how to access attributes afterwards?<br />
***** Use input value port to class modules?<br />
*** Wrap methods as modules? E.g., `ndarray.shape`.<br />
*** Function wrapping is almost a subset of class wrapping<br />
**** May be able to use the same parsing/execution methods<br />
** VTK's non-getter/setter methods could be wrapped as modules<br />
*** We could then remove extra logic in interpreter for keeping function order<br />
** Bokeh uses autogenerated docstrings<br />
*** We may be able to read the specification directly<br />
<br />
* [RR] [https://github.com/VisTrails/VisTrails/pull/1138 TensorFlow package]<br />
** Basic setup working, can execute the Mandelbrot example<br />
** Will autogenerate the operations<br />
<br />
=== November 4, 2015 ===<br />
'''Updates'''<br />
* [RR] Interpreter work requires a fix for [https://github.com/VisTrails/VisTrails/issues/1137 DB issue #1137]<br />
* [DK] Kitware's Resonant [http://www.kitware.com/source/home/post/173]<br />
** [https://github.com/girder/girder Girder]: data management system<br />
** [https://github.com/Kitware/romanesco Romanesco]: execution engine, uses [http://www.celeryproject.org/ Celery] for task management<br />
** Resonant Flow: web app for editing and executing workflows<br />
<br />
'''Items to Discuss'''<br />
* [TE] Added OSX Lion (10.7) VM on build machine using vagrant (seems ok with license?)<br />
** New builds works on Lion<br />
** Change the minimum requirement to 10.7?<br />
* [TE] Package wrapping<br />
* Added upgrade suggestions using 2 spec versions ([http://pastebin.com/30XUU8Gp example])<br />
** using name edit distance to find matches<br />
** Could use something better like port similarity for modules, and type similarity for ports<br />
** Should write python upgrade code?<br />
** Added spec to sklearn package<br />
*** Can now diff spec versions and keep the spec static<br />
*** There will be problems with downgrades and package versions<br />
<br />
=== October 29, 2015 ===<br />
'''Updates'''<br />
* [TE] nightly-build-mac fixed (Needed to approve xcode license)<br />
<br />
'''Items to Discuss'''<br />
* [TE] matplotlib wrapper<br />
** ported to general spec<br />
** Diff can be used on general spec<br />
*** Only needed minor changes<br />
*** Fixed indexing bug that corrupted specs<br />
*** Can now move diff tools to core/wrapper<br />
** Add new plots?<br />
*** New plots in mpl 1.3: eventplot<br />
*** New plots in mpl 1.4: angle_spectrum, magnitude_spectrum, phase_spectrum', violinplot<br />
*** No new plots in mpl 1.5 (from looking at boilerplate.py)<br />
<br />
=== October 21, 2015 ===<br />
'''Updates'''<br />
* [RR] Alexis has arrived, will be working with [RR] on a more efficient interpreter<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1133 New matplotlib package]<br />
** Supporting multiple package versions<br />
*** Check which version can be loaded [https://github.com/VisTrails/VisTrails/pull/1135 #1135]<br />
*** We cannot show version requirements in the list of packages because old packages are loaded when importing the codepath<br />
** Visual diff does not work well, but we can use the spec differ to see differences.<br />
** Unifying vtk and matplotlib wrapping specs (WIP)<br />
*** And porting matplotlibs spec differ to work on the general spec<br />
*** Spec differ can then be used on vtk and other packages in the future <br />
*** Will enable automatic upgrade generation<br />
* [RR] Rework interpreter<br />
** restore abstraction between interpreter/module code<br />
** build looping, streaming, caching into the interpreter<br />
** rework cache<br />
** look into parallelism once basic functionality is there<br />
<br />
=== October 14, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1133 New matplotlib package] (adding numpydoc parser)<br />
<br />
=== October 7, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://github.com/VisTrails/VisTrails/tree/fix-matplotlib-parser New matplotlib parser] (adding numpydoc parser)<br />
** Added numpydoc attribute parser for plots.<br />
*** TODO: Need better port spec reconciliation with call signature parser.<br />
*** We can create a general numpydoc parser, but attribute types are unique to matplotlib.<br />
** Needs package versioning<br />
*** New spec will not support matplotlib < v1.4 due to changed path to axes classes.<br />
*** Load spec version corresponding to installed matplotlib version?<br />
*** Need version downgrades?<br />
<br />
=== September 30, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Updating Matplotlib parser<br />
** Matplotlib docstring parser fails on numpy docstrings<br />
** I have [https://github.com/VisTrails/VisTrails/commit/eae8b0c16260810587e47c41cab53552cbb3cf0a added basic numpydoc parsing]<br />
*** Only used by a few docstrings so far<br />
*** Parsing uses many sources (class tables, signatures, ACCEPTS, method docstrings, definition parsing).<br />
** Caswell said they were thinking about moving to traitlets, but this is not ready yet<br />
<br />
=== September 23, 2015 ===<br />
'''Updates'''<br />
* [TE] VisTrails 2.2.3 released<br />
** Also have pushed to PyPI, binstar, etc.<br />
** Sourceforge vs. GitHub<br />
*** should be able to host releases on GitHub<br />
*** nightly binaries pushed to sf each night<br />
*** old binaries?<br />
* [TE] Re-wrapping MatPlotlib<br />
** Keep static generation - Docstrings are brittle beween matplotlib versions<br />
** Keep generating the executable classes - but create functions, not vistrails Modules <br />
*** Why is there so much patching?<br />
** Re-implemented Alternate PortSpec for InputPortSpec<br />
*** It will now inherit specs from parent (No need to reimplement)<br />
** Module upgrades?<br />
<br />
'''Items to Discuss'''<br />
<br />
=== September 16, 2015 ===<br />
'''Updates'''<br />
* [TE] Job Monitor tests and documentation done<br />
** OK to leave document package in packages directory?<br />
<br />
'''Items to Discuss'''<br />
* [TE] Library wrapping: How to do code patching? [https://docs.google.com/drawings/d/1Sh3eOXot6v2y880jHKQrqXqAwr-0aoqYligg3z8G8MY/edit?usp=sharing Wrapping diagram]<br />
** Insert code into module template (Old VTK code)<br />
** Patch the library that is called (New VTK code)<br />
** Somehow store code in spec and apply when executed?<br />
*** Executed code needs at a minimum access to inputs, outputs, current class<br />
*** Use standard input/output dict and obj reference that the code operates on?<br />
*** Can this be done while keeping the execution abstract?<br />
<br />
* Do new release now?<br />
<br />
=== September 9, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* New release?<br />
** [RR] wants reprounzip<br />
** tej changes & doc<br />
** job monitor & jobmixin fixes<br />
** mongodb<br />
** reprounzip<br />
** warning: MplFigure type is int<br />
** tabledata: add Reshape, DictoToTable & ListToTable, fix ListToTable with numpy arrays<br />
** PythonSource can have same name for an input & output<br />
** don't show spreadsheet at exit<br />
<br />
=== September 2, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Limit autosaves? [https://github.com/VisTrails/VisTrails/pull/1126 #1126]<br />
<br />
=== August 26, 2015 ===<br />
'''Updates'''<br />
* [TE] Added Job support to Parameter Explorations (Requested by Colin), and Mashups<br />
** specify job ids, need to specify different ids for parameter explorations since they have the same version id<br />
** how to deal with parameters passed in on command line<br />
* [General] provenance: should be creating a new version when we execute workflow with changed parameters?<br />
** currently, custom_params annotation stores this in provenance currently<br />
** [RR] API doesn't record provenance if passing in parameters: http://git.io/vsAA5<br />
'''Items to Discuss'''<br />
<br />
=== August 19, 2015 ===<br />
'''Updates'''<br />
* [RR] Improvements to tej, users' guide entry & example [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
* [RR] Internal docs [https://github.com/VisTrails/VisTrails/pull/1116 #1116]<br />
** On ReadTheDocs: [http://vistrails.readthedocs.org/ vistrails.readthedocs.org]<br />
* [TE] Working on Job Monitor documentation<br />
<br />
'''Items to Discuss'''<br />
* [RR] JobMixin and JobMonitor: stable now?<br />
** Definitely needs more tests<br />
* [TE] Stop testing VisTrails 2.0?<br />
<br />
=== August 12, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Internal docs [https://github.com/VisTrails/VisTrails/pull/1116 #1116]<br />
** On ReadTheDocs: [http://vistrails.readthedocs.org/ vistrails.readthedocs.org]<br />
* [RR] JobMixin and JobMonitor: stable now?<br />
** Definitely needs more tests<br />
* [RR] Improvements to tej, users' guide entry & example [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
* [RR] Build broken on Travis; because of IPython 4 released today? ([https://travis-ci.org/VisTrails/VisTrails/builds/75279795 build 992]; [https://github.com/VisTrails/VisTrails/issues/1123 #1123])<br />
<br />
=== August 5, 2015 ===<br />
'''Updates'''<br />
* [TE] Fixes to Jobs<br />
** Could not delete jobs<br />
** Could not run job in group<br />
** Job not reset when calling ModuleSuspended<br />
** Added deleting job from context menu<br />
* [RR] Writing documentation for everything [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
<br />
'''Items to Discuss'''<br />
<br />
=== July 29, 2015 ===<br />
'''Updates'''<br />
* RR still looking into new interpreter thing<br />
** Goal is to take out scheduling logic from Module so it can be split in multiple processes, and so that smarter strategies can be added in time<br />
** This means some work on packages<br />
** Spreadsheet can live in kernel process? Still some UI stuff that will take work (changing configuration, persistent archive's viewer, ...)<br />
* [TE] Problems running examples [https://github.com/VisTrails/VisTrails/pull/1111 #1111]<br />
** Testing of more examples requires additional packages on the test machines.<br />
** Fixed faulty line-ending in PythonSource:s failing on Python 2.6.<br />
** Test suite now testing SUDSWebServices (If web service is down, test suite will fail)<br />
** preferences.py test failed reloading 'dialogs' package, switched to using 'URL'.<br />
<br />
'''Items to Discuss'''<br />
<br />
=== July 22, 2015 ===<br />
'''Updates'''<br />
* [TE] mailing lists back online<br />
* [TE] Working on [https://github.com/VisTrails/VisTrails/issues/1107 #1107]<br />
* [RR] Working on ReproZip package <br />
<br />
'''Items to Discuss'''<br />
* [RR] MongoDB package [https://github.com/VisTrails/VisTrails/pull/1106 #1106]<br />
* [RR] Example for tej docs? [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
<br />
=== July 15, 2015 ===<br />
'''Updates'''<br />
* 2.2.2 released<br />
<br />
'''Items to Discuss'''<br />
* RR is considering executing everything in an IPython kernel (i.e. separate Python interpreter, like the one spawned for a notebook)<br />
** It's a separate process that we can restart without restarting the app/gui<br />
** We can isolate the execution environment (e.g. for the server)<br />
** We can run the whole thing remotely (if your data is elsewhere, just run VisTrails locally and the pipeline elsewhere)<br />
** Pipeline execution no longer makes the interface hang, it just makes the kernel hang (but that's fine)<br />
** We can use notebooks as modules (probably way nicer than the PythonSource module)<br />
** We can run multiple kernels so long as the ports carry things that are serializable<br />
*** meaning we can put the multithreaded-interpreter without all the hacky parts it has now<br />
*** we can run IPython kernels in all the languages IPython supports, [https://github.com/ipython/ipython/wiki/IPython-kernels-for-other-languages currently 46]<br />
<br />
=== July 8, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Buildbot github hook not working after IP address change<br />
* [TE] No reference to VistrailsApplication ([https://github.com/VisTrails/VisTrails/issues/1103 #1103])<br />
** [RR] Added PR [https://github.com/VisTrails/VisTrails/pull/1104 #1104]<br />
* [TE] Reopening VT file after saving with bundled subworkflow won't offer subworkflow upgrade ([https://github.com/VisTrails/VisTrails/issues/1102 #1102])<br />
** allow manual delete to fix right now<br />
** fix this on top of the use-uuid branch<br />
* [TE] Release VisTrails 2.2.1? ([https://github.com/VisTrails/VisTrails/blob/v2.2/CHANGELOG CHANGELOG])<br />
** [https://github.com/VisTrails/VisTrails/pull/1093 output-modules-use-global-dir]<br />
** [https://github.com/VisTrails/VisTrails/milestones/version%202.2 Current issues for 2.2]<br />
<br />
=== July 1, 2015 ===<br />
'''Updates'''<br />
* [TE] PROV fixed<br />
* [TE] Working on subworkflow issues<br />
* [RR] Considering reworking the controller (log vs exception problem, retained upgrades causing interferences) and also the interpreter (IPython?)<br />
<br />
'''Items to Discuss'''<br />
<br />
=== June 24, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* UV-CDAT<br />
* [TE] Can a cyclic workflow be valid? ([https://github.com/VisTrails/VisTrails/issues/1097 #1097])<br />
** focus on disabling the ability to create cyclic pipelines because more things break than just this with a cyclic pipeline<br />
* [RR] Relative paths ([https://github.com/VisTrails/VisTrails/pull/1057 #1057])<br />
** This interacts with the new bundle; how do handle packing files inside the VT bundle?<br />
<br />
=== June 17, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] current_version and reusing existing upgrades are broken ([https://github.com/VisTrails/VisTrails/issues/1095 ticket #1095])<br />
** Could be that export to PROV is using an unflushed upgrade pipeline?<br />
** current_version would then be correct.<br />
** It may work to flush the actions before exporting?<br />
<br />
=== June 10, 2015 ===<br />
'''Updates'''<br />
* DAT: fixed VTK issue on Linux and Mac<br />
* Still crashes on Windows. Need help! Reminder: this works in the VisTrails spreadsheet (QCellPresenter), although no widget get changed there during a drag<br />
** Is it a VTK bug?<br />
** Is it simply impossible to change widgets during the drag, should we do it a different way?<br />
** Did I miss something that is done in VisTrails but somehow not in DAT?<br />
** Low prio, UV-CDAT doesn't run on Windows anyway<br />
<br />
'''Items to Discuss'''<br />
* [TE] Executable Paper ([https://github.com/VisTrails/VisTrails/issues/1086 ticket #1088] [https://github.com/VisTrails/VisTrails/pull/1088 Updated pull request])<br />
* Single instance code and batch mode<br />
<br />
=== June 3, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Executable Paper ([https://github.com/VisTrails/VisTrails/issues/1086 ticket #1088] [https://github.com/VisTrails/VisTrails/pull/1088 pr])<br />
** Requires fixes to command line parameters, Output modules, and batch mode<br />
** How to test this<br />
** Updated missing/outdated flags<br />
** Fixed view issues when generating graphs<br />
* [TE] batch mode<br />
** SpreadsheetOutput not enabled in batch mode. Should we check is_running_gui instead?<br />
** Other instance setting flags from caller<br />
** Is graphsAsPdf replacing spreadsheetDumpPdf?<br />
** graphsAsPdf true by Default?<br />
** Batch mode executing by default (Not needed when generating graphs)<br />
** Re-added workflowInfo as withWorkflowInfo for writing graph and xml workflow<br />
** [DK] batch mode should be outputting to files or stdout, shouldn't always trigger SpreadsheetMode<br />
** execute flag, maybe make execution the default and allow a "--no-execute" if you only want to capture graphs, for example<br />
<br />
=== May 27, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] API question<br />
** http://lists.vistrails.org/pipermail/vistrails-users/2015-May/000357.html<br />
* Status of DAT<br />
** Can add VCS plot to DAT but need to work on configuration windows<br />
** Documentation for graphics templates for UV-CDAT/vcs?<br />
* Qt support?<br />
<br />
=== May 20, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Reviving the DAT, integrating scripting & porting UV-CDAT!<br />
** On [https://github.com/VisTrails/DAT GitHub] ([https://github.com/VisTrails/DAT/issues issues])<br />
** Merging 2 years of development taking longer than expected, but getting there. The plan is to get the patches in VisTrails and never fork again, we never want to get in UV-CDAT's situation (and don't need to).<br />
** VTK cell works fine on Linux but there was flickering on Mac & Windows before; still issues on Mac (Windows status unknown)<br />
** Can get a VCS plot soon (but will need VTK cell fix)<br />
** Integrate in UV-CDAT's build system (so we have cdms, VCS, ...) -> RR can do this, low priority<br />
** How do we integrate scripting?<br />
*** We want to be able to seamlessly make changes to a plot by changing Python code<br />
*** Define new plots by entering Python code without writing modules/packages?<br />
* [RR] UV-CDAT [https://github.com/UV-CDAT/uvcdat/issues/1275 needs] [https://github.com/VisTrails/VisTrails/pull/1073 #1073], please take a look<br />
* [RR] Subworkflow issues<br />
** [https://github.com/VisTrails/VisTrails/issues/1065 #1065]: missing attribute (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/1066 #1066]: groups in subworkflows were ignored when finding dependencies (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/725 #725]: "upgrade subworkflow" button not popping up (fixed); also, namespace issue<br />
** [https://github.com/VisTrails/VisTrails/issues/1071 #1071]: subworkflows don't go through upgrades<br />
* [RR] [https://github.com/VisTrails/VisTrails/issues/1074 #1074]: can't load and edit a single pipeline if upgrade happen<br />
<br />
=== May 13, 2015 ===<br />
'''Updates'''<br />
* [RR] Test skipping whitelist ([https://github.com/VisTrails/VisTrails/pull/1069 #1069]) -- low priority<br />
* [RR] Custom matplotlib modules can't be compatible with both 2.1 and 2.2 ([https://github.com/VisTrails/VisTrails/issues/1067 #1067]); should be fixed for ALPS ([https://github.com/VisTrails/VisTrails/issues/1070 #1070])<br />
<br />
'''Items to Discuss'''<br />
* [RR] UV-CDAT [https://github.com/UV-CDAT/uvcdat/issues/1275 needs] [https://github.com/VisTrails/VisTrails/pull/1073 #1073], please take a look<br />
* [RR] Subworkflow issues<br />
** [https://github.com/VisTrails/VisTrails/issues/1065 #1065]: missing attribute (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/1066 #1066]: groups in subworkflows were ignored when finding dependencies (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/725 #725]: "upgrade subworkflow" button not popping up (fixed); also, namespace issue<br />
** [https://github.com/VisTrails/VisTrails/issues/1071 #1071]: subworkflows don't go through upgrades<br />
* [RR] [https://github.com/VisTrails/VisTrails/issues/1074 #1074]: can't load and edit a single pipeline if upgrade happen<br />
<br />
=== May 7, 2015 ===<br />
'''Updates'''<br />
* [RR] Export/import workflow to Python working!<br />
<br />
'''Items to Discuss'''<br />
* [Claudio] UV-CDAT<br />
** The UV-CDAT project is the biggest user base of VisTrails<br />
** VisTrails package management provides a lot of friction towards people plugging in their code<br />
** Need to make it easy to integrate your random Python scripts in the system without having to deal with all the boilerplate, at least in the first step<br />
** [RR] argues that modules are still good; UV-CDAT shouldn't move towards a purely script-based backend<br />
** [RR] export/import with Python could reduce a lot of that friction by allowing 1) to edit workflow as Python 2) to open up boxes automatically if needed code doesn't match actual modules<br />
** ...<br />
<br />
=== April 29, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] <strike>Abstractions</strike> subworkflows status ([https://github.com/VisTrails/VisTrails/labels/subworkflows tickets])<br />
* [RR] matplotlib compatibility (2.1 & 2.2), [https://github.com/VisTrails/VisTrails/issues/1067 #1067]<br />
** RR to try and fix ALPS matplotlib modules<br />
<br />
=== April 22, 2015 ===<br />
'''Updates'''<br />
* 2.2.0 has been released!<br />
* Windows issue (via email from Ryan)<br />
** issue with manifest file (may be a new file in VTK6?)<br />
** Tommy has regenerated new Windows builds<br />
* Binaries, pypi, and conda released<br />
* [RR] Export as script<br />
* Python sources using VTK need to switch to SetInputData (users should be aware of this)<br />
<br />
=== April 15, 2015 ===<br />
'''Updates'''<br />
* Ready for 2.2.0 (apart from binary/deps issues)<br />
** Missing some libs (scikit-learn, tej, tdparser, SQLAlchemy+connectors<br />
** Windows: runvistrails.py is no longer used, so the PATH is wrong<br />
** Windows: pip is broken, but it probably wouldn't work anyway because of permission issues (disable this?)<br />
* Queries, upgrades and getPipeline() usage ([https://github.com/VisTrails/VisTrails/issues/1054 #1054])<br />
** Getting a pipeline with getPipeline() is not safe: it might return an invalid pipeline<br />
** This is used in many places throughout the code, like queries<br />
** Upgrading would require going through the controller, but that creates new actions<br />
* [TC] Avoid copying a module's output if it's used as input by exactly one downstream module ([https://github.com/VisTrails/VisTrails/issues/1060 #1060]) (useful for big numpy arrays you can update in-place)<br />
<br />
=== April 8, 2015 ===<br />
'''Updates'''<br />
* [DK] Merged RR's changes for output modules (1012 and 1013)<br />
** RR will merge remaining changes, then create v2.2 branch!<br />
<br />
'''Items to Discuss'''<br />
* BNL need numpy array to VTK image<br />
** Looks like VTK has some helpers for this<br />
** We will help if issues arise<br />
** Will contribute back to VisTrails package<br />
* Upgrade issue: [https://github.com/VisTrails/VisTrails/issues/1017 #1017]<br />
** Automatic upgrades should happen between versions of provided upgrades<br />
** Our existing upgrades work around this so it doesn't need to be 2.2.0<br />
* Corner-case VTK modules<br />
** No longer need VTKCell input port, so don't interfere with registry and API anymore<br />
** Still work weirdly, people probably shouldn't use them<br />
** But we have lots of clunky modules since we wrap the whole of VTK; some people might rely on this and know how to use them, let's keep them anyway<br />
** Ready for 2.2.0<br />
<br />
=== April 1, 2015 ===<br />
'''Updates'''<br />
* [RR] UV-CDAT: bugfixing for 2.2, long-term plans: implement scripting import/export in VisTrails, port to UV-CDAT<br />
* Possibly, try to move to regular VisTrails to use new features<br />
<br />
'''Items to Discuss'''<br />
* 2.2 release: nothing much is pending anymore, release next week?<br />
** [TE] vtkExporter classes [https://github.com/VisTrails/VisTrails/issues/1032 #1032]<br />
** [RR] Hiding upgrades in version tree might make it (or might be 2.2.1 so we can test it out) [https://github.com/VisTrails/VisTrails/issues/949 #949]<br />
** Output modules changes to go in<br />
* add note to documentation about order of parameters in VTK<br />
* add issue about exporter upgrades if not already there<br />
<br />
=== March 25, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* 2.2 release<br />
** Checklist on Github: https://github.com/VisTrails/VisTrails/wiki/2.2-Release-Checklist<br />
** Review output modules<br />
** RR has a couple more issues to fix<br />
** Ready to go -- sign app package for OSX? [https://github.com/VisTrails/VisTrails/issues/984 #984] We need access to the Apple Membership team<br />
<br />
=== March 18, 2015 ===<br />
'''Updates'''<br />
* [TE] VTK6 works<br />
<br />
'''Items to Discuss'''<br />
* [RR] Release v2.1.5 with backported tabledata?<br />
** MTA example needs updated tabledata (for JoinTables)<br />
** Google Maps package still not available<br />
* [RR] Work torward v2.2.0?<br />
** changes:<br />
*** new persistence<br />
*** API changes<br />
*** output module changes (upgrades?), maintain cells but try to upgrade<br />
*** not wrapping stuff<br />
*** VTK6? yes<br />
*** JobSubmission stuff?<br />
*** relabeling for upgrades [https://github.com/VisTrails/VisTrails/issues/949 #949]<br />
** makes sense, needs the tree view code to be updated, check selection<br />
** See [https://github.com/VisTrails/VisTrails/wiki/2.2-Release-Checklist 2.2 checklist]<br />
* Discussion of [https://github.com/VisTrails/VisTrails/issues/1016 #1016]<br />
** plumbing between outputs and output modes, how to define a mode that works for many outputs without writing for each output?<br />
<br />
=== March 11, 2015 ===<br />
'''Updates'''<br />
* [RR] persistent_archive done; merge? ([https://github.com/VisTrails/VisTrails/pull/755 #755])<br />
** note about the focus events for widgets<br />
** TE be aware of file_archive for future binaries that include persistent_archive<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/998 New VTK package] finished<br />
<br />
'''Items to Discuss'''<br />
* [RR] Unmark UV-CDAT/VisTrails as a fork of VisTrails/VisTrails? (see last week; decision needed)<br />
** RR email to JF about this<br />
** yes; email sent to Github<br />
* [RR] Switching order of output ports ([https://github.com/VisTrails/VisTrails/pull/1006 #1006])<br />
** added port specs are sorted at a separate spot (Module.*_port_specs properties) than those in the registry (which are sorted in the registry), but those two lists are just combined without respect to sort keys<br />
** need to determine whether the two lists should be merged or remain distinct<br />
** should make sure that order of input ports and output ports makes parallel connections for things with same order<br />
** DK suggests breaking backward compatibility here: workflows still run, can fix easily if a problem in an existing package.<br />
* [RR] Question about output modes ([https://github.com/VisTrails/VisTrails/pull/1007 #1007]), how to integrate in API ([https://github.com/VisTrails/VisTrails/pull/24 #24])<br />
** Should ImageFileMode be removed? ("image" is not a mode, "file" is)<br />
** ImageOutput missing?<br />
** Feel free to change how formats works<br />
* [TE] Test suite segfaults on Fedora 17 virtualbox. Install newer version? (Support ended in 2013)<br />
<br />
=== March 4, 2015 ===<br />
'''Updates'''<br />
* [TE] VTK wrapper<br />
** Works on VTK 5.10<br />
** Still need to test VTK 6<br />
** New general wrappers for python functions and classes <br />
<br />
'''Items to Discuss'''<br />
* [RR] Unmark UV-CDAT/VisTrails as a fork of VisTrails/VisTrails?<br />
** When [http://i.imgur.com/UTY2qfg.png filing a pull request] from UV-CDAT/VisTrails, VisTrails/VisTrails is selected by default<br />
** People keep forgetting to change the default ([https://github.com/VisTrails/VisTrails/pull/956 #956] [https://github.com/VisTrails/VisTrails/pull/968 #968] [https://github.com/VisTrails/VisTrails/pull/999 #999] [https://github.com/VisTrails/VisTrails/pull/1000 #1000] [https://github.com/VisTrails/VisTrails/pull/1003 #1003] [https://github.com/VisTrails/VisTrails/pull/1004 #1004] [https://github.com/VisTrails/VisTrails/pull/1005 #1005])<br />
** Only way to change that is to not have it marked as a fork of VisTrails/VisTrails<br />
** Github staff [https://gist.github.com/remram44/b364c24ef1e462ce6e2f can make that change for us]; should we do it?<br />
** Juliana: comments on this? visibility vs. convenience/annoyance for developers<br />
* [RR] What about [https://github.com/VisTrails/VisTrails/issues/991 #991]?<br />
<br />
=== February 25, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] vtkviewcell for infovis support, can we unify with VTKCell?<br />
** need to test this<br />
* [TE] vtk wrapping<br />
** Mostly finished<br />
** VTK 5.10 produce incorrect results with old wrapping<br />
*** Old wrapper is based mostly on VTK 4<br />
*** Most vtk_examples affected<br />
*** [http://www.vtk.org/Wiki/VTK/VTK_6_Migration/Replacement_of_SetInput]<br />
*** should be able to upgrade from SetInput to SetInputData (need to drop GetOutput and replace with self ports)<br />
*** can we change vtkInstance to just return self and not wrap things<br />
** terminator example not working under 5.8?<br />
* How does VTK wrapping fit into general wrapping framework?<br />
* [RR] new persistence package<br />
<br />
=== February 18, 2015 ===<br />
'''Updates'''<br />
* [RR] New VisTrails API and IPython integration ([https://github.com/VisTrails/VisTrails/pull/24 #24])<br />
<br />
'''Items to Discuss'''<br />
* [TE] VTK wrapping<br />
** [https://gist.github.com/rexissimus/16b74693545b5e7c1035 Benchmarking vtk package]<br />
*** Old: 24.7 seconds <br />
*** New: 10.5 seconds (Except first time that adds 8 sec)<br />
**** The parsing that calls is_abstract (that tries to instanciate all vtk classes) is now only run the first time.<br />
**** get_items_from_sigstring takes 2 seconds, maybe we can use a lookup dict for already computed sigstrings?<br />
** Now using a general python function wrapper<br />
*** VTK classes are wrapped into python function that does not depend on vistrails<br />
*** VTK functions can be executed without vistrails<br />
*** The spec maps functions into vistrails modules, but can also describe wrapping <br />
*** A general python function wrapper that supports <br />
**** kwarg inputs<br />
**** single, list, dict outputs<br />
**** callback for progress reporting<br />
**** temporary file generator for using FilePool<br />
**** optional output generation<br />
*** Creating specs:<br />
***** Create spec by hand<br />
***** Auto-create spec outline (TODO) and manually finish it<br />
***** Dynamically create spec (VTK)<br />
***** Implement documentation wrappers (Can use scikitlearn wrapper to wrap numpydoc) (TODO)<br />
***** Classes as bad functions needs to be wrapped in new functions before they are wrapped. This is different for each package.<br />
****** Classes is hard: Like VTK, and matplotlib. Scikit-learn does still not wrap classes<br />
***** Spec diffing and patching could be done using code from matplotlib.<br />
** Still needs upgrades from old VTK package<br />
*** Is it possible to dynamically wrap functions, e.g, you see a SetFunc and just remove the 'Set' prefix. Or do you need to create a complete mapping?<br />
<br />
=== February 11, 2015 ===<br />
'''Updates'''<br />
* Update from Friday's meeting<br />
** discussed VisTrails internals<br />
** discussed wrapping<br />
*** xml discussion, hard to modify because tied to db code<br />
*** TE has made it possible to add the schema-defined attributes to the intermediate representation<br />
*** higher-level operations on the port specs<br />
** make sure the simple case works<br />
*** [JF] take a simple package with documentation and figure out what the base case for wrapping is<br />
<br />
'''Items to Discuss'''<br />
* [TE] VTK wrapping<br />
** Dynamic loading works<br />
** Reading XML is fast enough, but [https://gist.github.com/rexissimus/8272d3c776ccfa115d82 unserializing data is slow]<br />
** Working on patterns for patching<br />
** Matplotlib has many advanced patterns like argument ordering, nested arguments, alternateSpecs, output types.<br />
*** Having all this in a general wrapper might confuse users?<br />
** [RR] Delay module (except for identifiers) until you need it---e.g. don't deal with port specs, etc. until necessary<br />
* Scripting Support [https://github.com/VisTrails/VisTrails/pull/950 #950]<br />
** [RR] Issue with getting code from modules<br />
** Design a simple solution<br />
** [JF] Couldn't you use modules as black boxes without conversion, just to call into modules/subworkflows easily from e.g. IPython?<br />
*** [RR] This is a job for the API, and a very separate use case. See [https://github.com/VisTrails/VisTrails/pull/24 #24]<br />
<br />
=== February 4, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* Wrapping<br />
** Format to use? Currently XML (like current matplotlib)<br />
*** JSON and YAML have simple "to python dictionary" methods<br />
*** But don't stream<br />
*** YAML a lot easier for humans<br />
** [DK] vtk-new-package also changes parameter names, creates enumerations<br />
*** intermediate schema needs to be extensible<br />
*** packages will want to store there specific infos for compute() method generation<br />
*** also might have specs-altering info, like matplotlib's alternateSpec<br />
** representation to code , registry already has schema for some aspects<br />
** [RR] We might want to see if Module subclasses can be created lazily<br />
*** no need to create all the classes just to register them in the registry and never actually use most of them<br />
*** future effort<br />
* [RR] Where should VisTrails packages live?<br />
** tej installs as 'vistrailspkg.tej', TE installed it as 'userpackages.tej'<br />
** Currently, standard packages are 'vistrails.packages.', user packages are 'userpackages.' and packages loaded through pkg_resources might be anything<br />
** [RR] Use 'vistrailspkg.' everywhere?<br />
** Long-term effort to simplify package distribution/installation (and have VisTrails get them automatically?)<br />
<br />
=== January 28, 2015 ===<br />
'''Updates'''<br />
* T. Caswell to come visit on Fri 6 to discuss wrapping work<br />
<br />
'''Items to Discuss'''<br />
* [TE] New VTK wrapping<br />
** Current code by DK seems a good deal faster<br />
** Generates XML that can be patched/tweaked, generates Python code from it, VisTrails only loads generated Python code<br />
** RR would rather have VisTrails load intermediate representation (= XML) directly, wants to make sure this is not slower<br />
** The goal is to turn the intermediate step into something generic that would be used for every wrapped package (vtk, numpy, matplotlib, sklearn, java) instead of each having its own<br />
** TC has its own code at [https://github.com/Nikea/VTTools/blob/master/vttools/wrap_lib.py github:VTTools] which parses numpy docstrings and generates modules, doesn't yet handle classes or persist anything<br />
* Web crawler<br />
** Right now, TE starts jobs for "start crawler", "stop crawler", "install classifier"<br />
** RR would rather have the crawling be a job as far as VisTrails and tej are concerned<br />
** The whole thing would be one pipeline: load examples, train classifier, start crawler [check for job, kill previous one, upload model, start processes], get snapshot, visualize<br />
*** Need some support in tej and job submission system: long-running jobs, stop a job (wait for it to finish?), restart a job even though results are cached<br />
<br />
=== January 21, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Unified wrapping method discussion [https://github.com/VisTrails/VisTrails/issues/991 #991]<br />
** TE to work on reusable method with intermediate representation, starting with VTK<br />
* [RR] Examples for scikit-learn: JF has an old example using Weka with parameter exploration (not currently in source tree)<br />
** AM's examples are enough<br />
* [AM] scikit-learn package is done, merge it in? [https://github.com/VisTrails/VisTrails/pull/955#issuecomment-70286788 #955]<br />
** RR will merge<br />
* [RR] What should copyright headers say? [https://github.com/VisTrails/VisTrails/pull/994#issuecomment-70173935 #994]<br />
** Let's keep everything in there: Utah/Poly/NYU<br />
<br />
=== January 14, 2015 ===<br />
'''Updates'''<br />
* [TE] Working on classifier<br />
* [RR] Scripting integration, work in progress<br />
<br />
'''Items to Discuss'''<br />
* [RR] Unified wrapping method discussion ([https://github.com/VisTrails/VisTrails/issues/991 #991])<br />
** Let's talk next week, [AM] and [DK] are not here<br />
<br />
=== January 7, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* make sure that we address critical issues, questions, and pending review branches in a timely manner<br />
* scripting support<br />
** [RR] no issues if we want to just keep annotations in the generated code to allow the link back to a workflow<br />
** [RR] can translate from workflow to script, working on script to workflow<br />
** will work for parameter value changes, structural changes require changes to the annotations<br />
** need to publish best practices here<br />
** would be cool to do looping in scripts (easier interface than with workflows)<br />
* notebook support (convert form notebook to workflow)<br />
** RR will sync with FC on this<br />
* Issue with console in built-from-scratch<br />
** [TC] iPython rearranged some of the completion stuff in 2.2 and 2.3<br />
** binary has old version of iPython -> 1.0.0, should we update?<br />
* [TC] automated wrapping of numpy and scipy<br />
** discovered a bunch of malformed documentation in numpy and scipy<br />
** has github repo for vistrails tools<br />
** example modules wrap a bunch of R stuff (not baked in, just how things are)<br />
** will be pushing wrapping logic up<br />
** port names forbidden (window and domain)<br />
** have an import hook to get from yaml directly to VisTrails Modules<br />
** should work for any python modules with well-formed numpy docstrings.<br />
* [Action] should make it clear in documentation that Constant now means serializable not that the value doesn't change (e.g. List)<br />
* [TC] might be interesting to try to build components of matplotlib and accumulate in figure (long-term project, but thinking about how this might work)<br />
* [TE] build and build scripts<br />
** completely automatic, buildbot<br />
** need to set the build machines for the environment we want for the binary<br />
** would virtualenv work here?<br />
** [TC] anaconda can pin versions, potential path to test different configurations<br />
** Q: upload nightly binary builds? A: makes sense, make sure they are well-labeled<br />
* sourceforge stats: e.g. http://sourceforge.net/projects/vistrails/files/vistrails/nightly/vistrails-src-nightly.tar.gz/stats/timeline?dates=2014-01-01+to+2015-01-07<br />
* package issues (see Remi's message)<br />
* [TE] Scope of tej<br />
** Support single ssh commands?<br />
** Queue can be used as a remote machine (crawler is using queue.call*)<br />
* SourceForge stats: http://sourceforge.net/projects/vistrails/files/vistrails/nightly/vistrails-src-nightly.tar.gz/stats/timeline?dates=2014-01-01+to+2015-01-07<br />
<br />
== Older meetings ==<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=12078Development2016-07-06T22:01:12Z<p>Remi: /* July 6, 2016 */</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2016 ==<br />
<br />
=== July 6, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
* [DK] Working on bundles<br />
<br />
'''Items to Discuss'''<br />
* [TE] Using vistrails API in scripting [https://gist.github.com/rexissimus/034d9c4dedc3299a079e84981dedd8c7 example 1 (API with VTK)] [https://gist.github.com/rexissimus/a4a9ca6c08d4058c64967d2b35415fd5 example 2 (DownloadFile only)]<br />
** Call module in script using vistrails API<br />
** Can be used when no export method is defined.<br />
* [TE] [https://github.com/VisTrails/VisTrails/commit/c76c109349769fee1280f85c22678380a2f9cd39 Server news bug] Critical?<br />
**[RR] No, unreleased<br />
* [DK] New bundle organization<br />
** BundleMapping contains individual mappings from raw objects to BundleObj<br />
** Must be separate in order to have in-memory, separate from serializer settings<br />
<br />
=== June 29, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as python script<br />
** Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) [https://gist.github.com/rexissimus/16caf5200802e150296f62a0c7844c5a example]<br />
** Preludes (imports) still need improvements<br />
** Works with vtkInteractionHandler (clipping plane)<br />
** Resulting clipping plane is incorrect? (supposed to hide one side of the plane)<br />
** vtkRendererOutput Uses VTKRenderWindow as output<br />
*** Should support different output modes?<br />
*** Does not work with multiple outputs (Next cell is displayed when current is closed)<br />
*** Uses wrapped vtkInstance (as vvtk)<br />
**** Create new patch system with code that can be used both in vistrails and in a python script?<br />
<br />
=== June 22, 2016 ===<br />
<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fix for upgraded analogies (PR [https://github.com/VisTrails/VisTrails/pull/1177 #1177])<br />
** Uses module location to find module remap (The best we can do right now?)<br />
** Also fixes remapped port names for connections to existing modules <br />
** May still fail on complex upgrades like adding a function on an upgraded module that has a different port name<br />
<br />
* [TE] Working on exporting wrapped libraries (VTK)<br />
<br />
=== June 15, 2016 ===<br />
<br />
'''Updates'''<br />
* [TE] Added depth logic to python script exporter ([https://github.com/VisTrails/VisTrails/commit/50f220e4f6abe5a7bb8689f73f456f43df2dbc12 full commit])<br />
** Fully implements connection merging and list looping<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Making analogy work on upgraded workflows ([https://github.com/VisTrails/VisTrails/issues/1175 #1175])<br />
** Analogy only works We need to use the non-upgraded action chain<br />
** But pipelines need to be valid (matcher uses module portspecs)<br />
** Tried to get upgrade remap from upgrade action, but this seems impossible<br />
** Use module location instead?<br />
*** Location should be identical for one-to-one module upgrades<br />
<br />
=== June 1, 2016 ===<br />
<br />
'''Updates'''<br />
* VisTrails 2.2.4 announced (finally)<br />
** 5 survey replies<br />
** 47 downloads last week<br />
<br />
'''Items to Discuss'''<br />
* [TE] Export VTK as script (python-wrapper)<br />
** to_python_script<br />
*** needs the pipeline (or at least the connections)<br />
**** Should not generate code for any possible VTK method (exporter detects which ports is used in the source)<br />
**** VTK needs to know connection order<br />
**** Adding functions directly to the script results in nicer code<br />
** Add support for multiple input connections<br />
** Add support for list looping<br />
<br />
=== May 25, 2016 ===<br />
<br />
'''Updates'''<br />
* Mailing-list fixed<br />
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])<br />
<br />
'''Items to Discuss'''<br />
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)<br />
* [TE] Union ports<br />
** Show union ports in pipeline (DEMO)<br />
<br />
=== May 18, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] mailing list still down - Julio working on it<br />
* [TE] Improved union port<br />
** New schema 1.0.5 adds portitem.union string attribute<br />
** Works with pipeline view, parameter exploration view, and mashup view<br />
** Should work well for matplotlib<br />
** Use with VTK ports?<br />
*** It uses [item_1,item_2] style ports usually with different tuple length.<br />
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]<br />
<br />
=== May 11, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?<br />
*** Shows single port for all ports with same sort_key<br />
*** Select specific type when adding/showing<br />
*** Only one type should be visible on the module at any time<br />
*** Making tuple parameters union types is probably too complicated<br />
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly) <br />
<br />
* [TE] Feature suggestion: Let connections connect to functions/parameters<br />
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters<br />
** Use separate views for ports and functions (like before)<br />
*** Allow rearranging of functions<br />
** Pros:<br />
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.<br />
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.<br />
**** Makes List module simpler<br />
*** Connect to parameter in tuple<br />
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds. <br />
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.<br />
** Cons:<br />
*** Interface more complicated and crowded<br />
*** More difficult for users to grasp?<br />
*** No immediate use case<br />
*** Makes no union type for tuple parameters a bigger problem.<br />
<br />
* [TE] Publish user survey?<br />
** No response from vistrails release yet<br />
** Not announced on vistrails-users yet.<br />
<br />
=== May 4, 2016 ===<br />
'''Updates'''<br />
* VisTrails 2.2.4 released<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
<br />
=== April 27, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Wrapping Bokeh<br />
** New method type "operation"<br />
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output<br />
**** Glyph sets function name and arguments as output<br />
***** Not serializable, but should always be computed together with the subject of the operation.<br />
**** Figure applies Glyph operation on itself<br />
*** Useful for matplotlib?<br />
**** matplotlib currently find actors by searching upstream?<br />
**** But this is more difficult since matplotlib is not function-based.<br />
** New "union" port type<br />
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value<br />
**** Uses only first value in tuple<br />
**** Bokeh have many properties that support (list, string, Float)<br />
*** Does not work with depth=1 types (e.g. Float of depth 1)<br />
**** Turn Depth 1 types with known type into lists?<br />
<br />
* [TE] New bugfix release<br />
** Finalize survey<br />
<br />
=== April 20, 2016 ===<br />
'''Updates'''<br />
* Merged upgrade fixes<br />
** Now has infinite loop test<br />
** loop test now correctly loops 5 times<br />
*** remap_module used wrong (current) package version when creating ModuleDescriptor<br />
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])<br />
<br />
'''Items to Discuss'''<br />
* New bugfix release<br />
** Merge usagestats<br />
** Need to finalize survey<br />
<br />
=== April 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?<br />
<br />
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])<br />
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]<br />
** Cherry-pick test to preserve source?<br />
** Need maxPipelineFixAttempts option?<br />
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.<br />
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?<br />
*** Would need to be moved into handle_invalid_pipeline<br />
<br />
=== April 6, 2016 ===<br />
'''Updates'''<br />
* [TE] Query View improvements<br />
** Done except for displaying "Version Info" and #1164<br />
* [RR] Working on finishing new interpreter (finally!)<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Try latest upgrade first, then previous.<br />
** If all fail, select latest upgrade?<br />
<br />
=== March 30, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Query View improvements<br />
** Unified do_version_switch and get_upgrades as validate_version()<br />
** Problems with query view using its own controller<br />
*** Fixed module info not being displayed for query results<br />
*** Still no version info for query results<br />
*** Still an issue with version query view not being updated<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== 2015 ==<br />
<br />
=== December 30, 2015 ===<br />
'''Updates'''<br />
* Bokeh package finished<br />
** Can output to Browser/IPython/Spreadsheet (Limited functionality in spreadsheet)<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Bokeh needs QWebEngine (Chromium) in Qt5 which requires PyQt5<br />
*** Many plotting libraries only work in browsers<br />
*** But Chromium may be a big and controversial dependency<br />
*** Or create a browser version of the spreadsheet?<br />
** Some users have requested Python 3<br />
** Not backwards compatible?<br />
** Target for VisTrails 3.0 with new interpreter?<br />
<br />
=== December 16, 2015 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Users list question<br />
* [TE] Wrapping Bokeh<br />
** Added bokeh property parser<br />
*** complements numpydoc parser<br />
*** Not all attributes are available, partially because I am using a pre-release version.<br />
** Wrapped modules<br />
*** bokeh.plotting finished (Backwards like matplotlib: Create plot then draw on it)<br />
*** bokeh.charts mostly done (Missing arguments)<br />
** QWebView mostly works (but no zooming or buttons)<br />
*** It would be better to use external browser or QWebEngineView<br />
** Added improvements to wrapper<br />
*** Generalized port translations (Color/Path to native types)<br />
*** Can use multiple docstring/property parsers simultaneously<br />
<br />
=== December 9, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Looking into wrapper<br />
** Will port TensorFlow<br />
** Doing a pass on the wrapper code & doc<br />
*** UX improvements<br />
** Factorize high-level docstring parsers into core.wrapper? (sphinx, numpy, google)<br />
** Problem with name vs module_name (fails sklearn tests because class_by_name() is broken)<br />
** Looking into compressing specs, lazy registration of modules<br />
<br />
=== December 2, 2015 ===<br />
'''Updates'''<br />
* [TE] Package wrapping<br />
** Wrapped most of numpy/scipy using __all__<br />
** Added class attribute and method access for classes (Can add to class or to separate inspector module)<br />
** TODO: Document wrapping procedure<br />
<br />
'''Items to Discuss'''<br />
<br />
=== November 25, 2015 ===<br />
'''Updates'''<br />
* [RR] New interpreter is coming along<br />
** Everything is a stream<br />
** Task system, eventually work-stealing parallelism<br />
** Streams no longer advance in lock-step<br />
*** Allows for constructions like filter, join, sorted-merge<br />
** depth>1 still to be tested, no current plan to allow this through the module interface<br />
*** Through looping though, can do streams of streams<br />
<br />
'''Items to Discuss'''<br />
* [TE] Package wrapping<br />
** Wrapped most numpy functions (no polynomials)<br />
** and half of scipy classes and functions<br />
** Added functions manually from documentation (no parseable lists?)<br />
** Focusing on data flow constructs (no property/method access for classes)<br />
*** Using spec diff (empty for now) and dynamic parser (stores spec in .vistrails/)<br />
** Python class/function wrapper now stable<br />
** Add google doc parser?<br />
*** Is there a parser available?<br />
<br />
=== November 18, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Package wrapping<br />
** Numpy class wrapping works<br />
*** Basic type is List, since most types are array_likes.<br />
*** TODO: More classes and functions<br />
*** Some operations are in-place and some aren't, and docstring not super clear<br />
*** [RR] It seems only methods mutate, and most have a numpy.xx function equivalent; just manually go through the methods and remove the mutating ones?<br />
** Class wrapper is modular<br />
*** Docstring parser and type string parser can be customized<br />
*** Classes can have optional attribute/method ports<br />
**** Inspectors and attribute/method modules can be created separately<br />
** Use PythonCalc as an example<br />
*** Can be re-implemented as a function with a parseable docstring<br />
*** [RR] PythonCalc is just an example, and it might actually make sense to get rid of it, or do a proper math package with scalar operations (as separate modules, no combobox)<br />
** [RR] TensorFlow to use wrapping as well<br />
*** Very simple (only types 'tensor' & 'variable')<br />
<br />
=== November 11, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Package wrapping<br />
** Wrapping numpy's classes using numpydoc (ndarray)<br />
*** Wrapping constructor arguments, attribute getters/setters as module ports<br />
**** Then how to access attributes afterwards?<br />
***** Use input value port to class modules?<br />
*** Wrap methods as modules? E.g., `ndarray.shape`.<br />
*** Function wrapping is almost a subset of class wrapping<br />
**** May be able to use the same parsing/execution methods<br />
** VTK's non-getter/setter methods could be wrapped as modules<br />
*** We could then remove extra logic in interpreter for keeping function order<br />
** Bokeh uses autogenerated docstrings<br />
*** We may be able to read the specification directly<br />
<br />
* [RR] [https://github.com/VisTrails/VisTrails/pull/1138 TensorFlow package]<br />
** Basic setup working, can execute the Mandelbrot example<br />
** Will autogenerate the operations<br />
<br />
=== November 4, 2015 ===<br />
'''Updates'''<br />
* [RR] Interpreter work requires a fix for [https://github.com/VisTrails/VisTrails/issues/1137 DB issue #1137]<br />
* [DK] Kitware's Resonant [http://www.kitware.com/source/home/post/173]<br />
** [https://github.com/girder/girder Girder]: data management system<br />
** [https://github.com/Kitware/romanesco Romanesco]: execution engine, uses [http://www.celeryproject.org/ Celery] for task management<br />
** Resonant Flow: web app for editing and executing workflows<br />
<br />
'''Items to Discuss'''<br />
* [TE] Added OSX Lion (10.7) VM on build machine using vagrant (seems ok with license?)<br />
** New builds works on Lion<br />
** Change the minimum requirement to 10.7?<br />
* [TE] Package wrapping<br />
* Added upgrade suggestions using 2 spec versions ([http://pastebin.com/30XUU8Gp example])<br />
** using name edit distance to find matches<br />
** Could use something better like port similarity for modules, and type similarity for ports<br />
** Should write python upgrade code?<br />
** Added spec to sklearn package<br />
*** Can now diff spec versions and keep the spec static<br />
*** There will be problems with downgrades and package versions<br />
<br />
=== October 29, 2015 ===<br />
'''Updates'''<br />
* [TE] nightly-build-mac fixed (Needed to approve xcode license)<br />
<br />
'''Items to Discuss'''<br />
* [TE] matplotlib wrapper<br />
** ported to general spec<br />
** Diff can be used on general spec<br />
*** Only needed minor changes<br />
*** Fixed indexing bug that corrupted specs<br />
*** Can now move diff tools to core/wrapper<br />
** Add new plots?<br />
*** New plots in mpl 1.3: eventplot<br />
*** New plots in mpl 1.4: angle_spectrum, magnitude_spectrum, phase_spectrum', violinplot<br />
*** No new plots in mpl 1.5 (from looking at boilerplate.py)<br />
<br />
=== October 21, 2015 ===<br />
'''Updates'''<br />
* [RR] Alexis has arrived, will be working with [RR] on a more efficient interpreter<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1133 New matplotlib package]<br />
** Supporting multiple package versions<br />
*** Check which version can be loaded [https://github.com/VisTrails/VisTrails/pull/1135 #1135]<br />
*** We cannot show version requirements in the list of packages because old packages are loaded when importing the codepath<br />
** Visual diff does not work well, but we can use the spec differ to see differences.<br />
** Unifying vtk and matplotlib wrapping specs (WIP)<br />
*** And porting matplotlibs spec differ to work on the general spec<br />
*** Spec differ can then be used on vtk and other packages in the future <br />
*** Will enable automatic upgrade generation<br />
* [RR] Rework interpreter<br />
** restore abstraction between interpreter/module code<br />
** build looping, streaming, caching into the interpreter<br />
** rework cache<br />
** look into parallelism once basic functionality is there<br />
<br />
=== October 14, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1133 New matplotlib package] (adding numpydoc parser)<br />
<br />
=== October 7, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://github.com/VisTrails/VisTrails/tree/fix-matplotlib-parser New matplotlib parser] (adding numpydoc parser)<br />
** Added numpydoc attribute parser for plots.<br />
*** TODO: Need better port spec reconciliation with call signature parser.<br />
*** We can create a general numpydoc parser, but attribute types are unique to matplotlib.<br />
** Needs package versioning<br />
*** New spec will not support matplotlib < v1.4 due to changed path to axes classes.<br />
*** Load spec version corresponding to installed matplotlib version?<br />
*** Need version downgrades?<br />
<br />
=== September 30, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Updating Matplotlib parser<br />
** Matplotlib docstring parser fails on numpy docstrings<br />
** I have [https://github.com/VisTrails/VisTrails/commit/eae8b0c16260810587e47c41cab53552cbb3cf0a added basic numpydoc parsing]<br />
*** Only used by a few docstrings so far<br />
*** Parsing uses many sources (class tables, signatures, ACCEPTS, method docstrings, definition parsing).<br />
** Caswell said they were thinking about moving to traitlets, but this is not ready yet<br />
<br />
=== September 23, 2015 ===<br />
'''Updates'''<br />
* [TE] VisTrails 2.2.3 released<br />
** Also have pushed to PyPI, binstar, etc.<br />
** Sourceforge vs. GitHub<br />
*** should be able to host releases on GitHub<br />
*** nightly binaries pushed to sf each night<br />
*** old binaries?<br />
* [TE] Re-wrapping MatPlotlib<br />
** Keep static generation - Docstrings are brittle beween matplotlib versions<br />
** Keep generating the executable classes - but create functions, not vistrails Modules <br />
*** Why is there so much patching?<br />
** Re-implemented Alternate PortSpec for InputPortSpec<br />
*** It will now inherit specs from parent (No need to reimplement)<br />
** Module upgrades?<br />
<br />
'''Items to Discuss'''<br />
<br />
=== September 16, 2015 ===<br />
'''Updates'''<br />
* [TE] Job Monitor tests and documentation done<br />
** OK to leave document package in packages directory?<br />
<br />
'''Items to Discuss'''<br />
* [TE] Library wrapping: How to do code patching? [https://docs.google.com/drawings/d/1Sh3eOXot6v2y880jHKQrqXqAwr-0aoqYligg3z8G8MY/edit?usp=sharing Wrapping diagram]<br />
** Insert code into module template (Old VTK code)<br />
** Patch the library that is called (New VTK code)<br />
** Somehow store code in spec and apply when executed?<br />
*** Executed code needs at a minimum access to inputs, outputs, current class<br />
*** Use standard input/output dict and obj reference that the code operates on?<br />
*** Can this be done while keeping the execution abstract?<br />
<br />
* Do new release now?<br />
<br />
=== September 9, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* New release?<br />
** [RR] wants reprounzip<br />
** tej changes & doc<br />
** job monitor & jobmixin fixes<br />
** mongodb<br />
** reprounzip<br />
** warning: MplFigure type is int<br />
** tabledata: add Reshape, DictoToTable & ListToTable, fix ListToTable with numpy arrays<br />
** PythonSource can have same name for an input & output<br />
** don't show spreadsheet at exit<br />
<br />
=== September 2, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Limit autosaves? [https://github.com/VisTrails/VisTrails/pull/1126 #1126]<br />
<br />
=== August 26, 2015 ===<br />
'''Updates'''<br />
* [TE] Added Job support to Parameter Explorations (Requested by Colin), and Mashups<br />
** specify job ids, need to specify different ids for parameter explorations since they have the same version id<br />
** how to deal with parameters passed in on command line<br />
* [General] provenance: should be creating a new version when we execute workflow with changed parameters?<br />
** currently, custom_params annotation stores this in provenance currently<br />
** [RR] API doesn't record provenance if passing in parameters: http://git.io/vsAA5<br />
'''Items to Discuss'''<br />
<br />
=== August 19, 2015 ===<br />
'''Updates'''<br />
* [RR] Improvements to tej, users' guide entry & example [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
* [RR] Internal docs [https://github.com/VisTrails/VisTrails/pull/1116 #1116]<br />
** On ReadTheDocs: [http://vistrails.readthedocs.org/ vistrails.readthedocs.org]<br />
* [TE] Working on Job Monitor documentation<br />
<br />
'''Items to Discuss'''<br />
* [RR] JobMixin and JobMonitor: stable now?<br />
** Definitely needs more tests<br />
* [TE] Stop testing VisTrails 2.0?<br />
<br />
=== August 12, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Internal docs [https://github.com/VisTrails/VisTrails/pull/1116 #1116]<br />
** On ReadTheDocs: [http://vistrails.readthedocs.org/ vistrails.readthedocs.org]<br />
* [RR] JobMixin and JobMonitor: stable now?<br />
** Definitely needs more tests<br />
* [RR] Improvements to tej, users' guide entry & example [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
* [RR] Build broken on Travis; because of IPython 4 released today? ([https://travis-ci.org/VisTrails/VisTrails/builds/75279795 build 992]; [https://github.com/VisTrails/VisTrails/issues/1123 #1123])<br />
<br />
=== August 5, 2015 ===<br />
'''Updates'''<br />
* [TE] Fixes to Jobs<br />
** Could not delete jobs<br />
** Could not run job in group<br />
** Job not reset when calling ModuleSuspended<br />
** Added deleting job from context menu<br />
* [RR] Writing documentation for everything [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
<br />
'''Items to Discuss'''<br />
<br />
=== July 29, 2015 ===<br />
'''Updates'''<br />
* RR still looking into new interpreter thing<br />
** Goal is to take out scheduling logic from Module so it can be split in multiple processes, and so that smarter strategies can be added in time<br />
** This means some work on packages<br />
** Spreadsheet can live in kernel process? Still some UI stuff that will take work (changing configuration, persistent archive's viewer, ...)<br />
* [TE] Problems running examples [https://github.com/VisTrails/VisTrails/pull/1111 #1111]<br />
** Testing of more examples requires additional packages on the test machines.<br />
** Fixed faulty line-ending in PythonSource:s failing on Python 2.6.<br />
** Test suite now testing SUDSWebServices (If web service is down, test suite will fail)<br />
** preferences.py test failed reloading 'dialogs' package, switched to using 'URL'.<br />
<br />
'''Items to Discuss'''<br />
<br />
=== July 22, 2015 ===<br />
'''Updates'''<br />
* [TE] mailing lists back online<br />
* [TE] Working on [https://github.com/VisTrails/VisTrails/issues/1107 #1107]<br />
* [RR] Working on ReproZip package <br />
<br />
'''Items to Discuss'''<br />
* [RR] MongoDB package [https://github.com/VisTrails/VisTrails/pull/1106 #1106]<br />
* [RR] Example for tej docs? [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
<br />
=== July 15, 2015 ===<br />
'''Updates'''<br />
* 2.2.2 released<br />
<br />
'''Items to Discuss'''<br />
* RR is considering executing everything in an IPython kernel (i.e. separate Python interpreter, like the one spawned for a notebook)<br />
** It's a separate process that we can restart without restarting the app/gui<br />
** We can isolate the execution environment (e.g. for the server)<br />
** We can run the whole thing remotely (if your data is elsewhere, just run VisTrails locally and the pipeline elsewhere)<br />
** Pipeline execution no longer makes the interface hang, it just makes the kernel hang (but that's fine)<br />
** We can use notebooks as modules (probably way nicer than the PythonSource module)<br />
** We can run multiple kernels so long as the ports carry things that are serializable<br />
*** meaning we can put the multithreaded-interpreter without all the hacky parts it has now<br />
*** we can run IPython kernels in all the languages IPython supports, [https://github.com/ipython/ipython/wiki/IPython-kernels-for-other-languages currently 46]<br />
<br />
=== July 8, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Buildbot github hook not working after IP address change<br />
* [TE] No reference to VistrailsApplication ([https://github.com/VisTrails/VisTrails/issues/1103 #1103])<br />
** [RR] Added PR [https://github.com/VisTrails/VisTrails/pull/1104 #1104]<br />
* [TE] Reopening VT file after saving with bundled subworkflow won't offer subworkflow upgrade ([https://github.com/VisTrails/VisTrails/issues/1102 #1102])<br />
** allow manual delete to fix right now<br />
** fix this on top of the use-uuid branch<br />
* [TE] Release VisTrails 2.2.1? ([https://github.com/VisTrails/VisTrails/blob/v2.2/CHANGELOG CHANGELOG])<br />
** [https://github.com/VisTrails/VisTrails/pull/1093 output-modules-use-global-dir]<br />
** [https://github.com/VisTrails/VisTrails/milestones/version%202.2 Current issues for 2.2]<br />
<br />
=== July 1, 2015 ===<br />
'''Updates'''<br />
* [TE] PROV fixed<br />
* [TE] Working on subworkflow issues<br />
* [RR] Considering reworking the controller (log vs exception problem, retained upgrades causing interferences) and also the interpreter (IPython?)<br />
<br />
'''Items to Discuss'''<br />
<br />
=== June 24, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* UV-CDAT<br />
* [TE] Can a cyclic workflow be valid? ([https://github.com/VisTrails/VisTrails/issues/1097 #1097])<br />
** focus on disabling the ability to create cyclic pipelines because more things break than just this with a cyclic pipeline<br />
* [RR] Relative paths ([https://github.com/VisTrails/VisTrails/pull/1057 #1057])<br />
** This interacts with the new bundle; how do handle packing files inside the VT bundle?<br />
<br />
=== June 17, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] current_version and reusing existing upgrades are broken ([https://github.com/VisTrails/VisTrails/issues/1095 ticket #1095])<br />
** Could be that export to PROV is using an unflushed upgrade pipeline?<br />
** current_version would then be correct.<br />
** It may work to flush the actions before exporting?<br />
<br />
=== June 10, 2015 ===<br />
'''Updates'''<br />
* DAT: fixed VTK issue on Linux and Mac<br />
* Still crashes on Windows. Need help! Reminder: this works in the VisTrails spreadsheet (QCellPresenter), although no widget get changed there during a drag<br />
** Is it a VTK bug?<br />
** Is it simply impossible to change widgets during the drag, should we do it a different way?<br />
** Did I miss something that is done in VisTrails but somehow not in DAT?<br />
** Low prio, UV-CDAT doesn't run on Windows anyway<br />
<br />
'''Items to Discuss'''<br />
* [TE] Executable Paper ([https://github.com/VisTrails/VisTrails/issues/1086 ticket #1088] [https://github.com/VisTrails/VisTrails/pull/1088 Updated pull request])<br />
* Single instance code and batch mode<br />
<br />
=== June 3, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Executable Paper ([https://github.com/VisTrails/VisTrails/issues/1086 ticket #1088] [https://github.com/VisTrails/VisTrails/pull/1088 pr])<br />
** Requires fixes to command line parameters, Output modules, and batch mode<br />
** How to test this<br />
** Updated missing/outdated flags<br />
** Fixed view issues when generating graphs<br />
* [TE] batch mode<br />
** SpreadsheetOutput not enabled in batch mode. Should we check is_running_gui instead?<br />
** Other instance setting flags from caller<br />
** Is graphsAsPdf replacing spreadsheetDumpPdf?<br />
** graphsAsPdf true by Default?<br />
** Batch mode executing by default (Not needed when generating graphs)<br />
** Re-added workflowInfo as withWorkflowInfo for writing graph and xml workflow<br />
** [DK] batch mode should be outputting to files or stdout, shouldn't always trigger SpreadsheetMode<br />
** execute flag, maybe make execution the default and allow a "--no-execute" if you only want to capture graphs, for example<br />
<br />
=== May 27, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] API question<br />
** http://lists.vistrails.org/pipermail/vistrails-users/2015-May/000357.html<br />
* Status of DAT<br />
** Can add VCS plot to DAT but need to work on configuration windows<br />
** Documentation for graphics templates for UV-CDAT/vcs?<br />
* Qt support?<br />
<br />
=== May 20, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Reviving the DAT, integrating scripting & porting UV-CDAT!<br />
** On [https://github.com/VisTrails/DAT GitHub] ([https://github.com/VisTrails/DAT/issues issues])<br />
** Merging 2 years of development taking longer than expected, but getting there. The plan is to get the patches in VisTrails and never fork again, we never want to get in UV-CDAT's situation (and don't need to).<br />
** VTK cell works fine on Linux but there was flickering on Mac & Windows before; still issues on Mac (Windows status unknown)<br />
** Can get a VCS plot soon (but will need VTK cell fix)<br />
** Integrate in UV-CDAT's build system (so we have cdms, VCS, ...) -> RR can do this, low priority<br />
** How do we integrate scripting?<br />
*** We want to be able to seamlessly make changes to a plot by changing Python code<br />
*** Define new plots by entering Python code without writing modules/packages?<br />
* [RR] UV-CDAT [https://github.com/UV-CDAT/uvcdat/issues/1275 needs] [https://github.com/VisTrails/VisTrails/pull/1073 #1073], please take a look<br />
* [RR] Subworkflow issues<br />
** [https://github.com/VisTrails/VisTrails/issues/1065 #1065]: missing attribute (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/1066 #1066]: groups in subworkflows were ignored when finding dependencies (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/725 #725]: "upgrade subworkflow" button not popping up (fixed); also, namespace issue<br />
** [https://github.com/VisTrails/VisTrails/issues/1071 #1071]: subworkflows don't go through upgrades<br />
* [RR] [https://github.com/VisTrails/VisTrails/issues/1074 #1074]: can't load and edit a single pipeline if upgrade happen<br />
<br />
=== May 13, 2015 ===<br />
'''Updates'''<br />
* [RR] Test skipping whitelist ([https://github.com/VisTrails/VisTrails/pull/1069 #1069]) -- low priority<br />
* [RR] Custom matplotlib modules can't be compatible with both 2.1 and 2.2 ([https://github.com/VisTrails/VisTrails/issues/1067 #1067]); should be fixed for ALPS ([https://github.com/VisTrails/VisTrails/issues/1070 #1070])<br />
<br />
'''Items to Discuss'''<br />
* [RR] UV-CDAT [https://github.com/UV-CDAT/uvcdat/issues/1275 needs] [https://github.com/VisTrails/VisTrails/pull/1073 #1073], please take a look<br />
* [RR] Subworkflow issues<br />
** [https://github.com/VisTrails/VisTrails/issues/1065 #1065]: missing attribute (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/1066 #1066]: groups in subworkflows were ignored when finding dependencies (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/725 #725]: "upgrade subworkflow" button not popping up (fixed); also, namespace issue<br />
** [https://github.com/VisTrails/VisTrails/issues/1071 #1071]: subworkflows don't go through upgrades<br />
* [RR] [https://github.com/VisTrails/VisTrails/issues/1074 #1074]: can't load and edit a single pipeline if upgrade happen<br />
<br />
=== May 7, 2015 ===<br />
'''Updates'''<br />
* [RR] Export/import workflow to Python working!<br />
<br />
'''Items to Discuss'''<br />
* [Claudio] UV-CDAT<br />
** The UV-CDAT project is the biggest user base of VisTrails<br />
** VisTrails package management provides a lot of friction towards people plugging in their code<br />
** Need to make it easy to integrate your random Python scripts in the system without having to deal with all the boilerplate, at least in the first step<br />
** [RR] argues that modules are still good; UV-CDAT shouldn't move towards a purely script-based backend<br />
** [RR] export/import with Python could reduce a lot of that friction by allowing 1) to edit workflow as Python 2) to open up boxes automatically if needed code doesn't match actual modules<br />
** ...<br />
<br />
=== April 29, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] <strike>Abstractions</strike> subworkflows status ([https://github.com/VisTrails/VisTrails/labels/subworkflows tickets])<br />
* [RR] matplotlib compatibility (2.1 & 2.2), [https://github.com/VisTrails/VisTrails/issues/1067 #1067]<br />
** RR to try and fix ALPS matplotlib modules<br />
<br />
=== April 22, 2015 ===<br />
'''Updates'''<br />
* 2.2.0 has been released!<br />
* Windows issue (via email from Ryan)<br />
** issue with manifest file (may be a new file in VTK6?)<br />
** Tommy has regenerated new Windows builds<br />
* Binaries, pypi, and conda released<br />
* [RR] Export as script<br />
* Python sources using VTK need to switch to SetInputData (users should be aware of this)<br />
<br />
=== April 15, 2015 ===<br />
'''Updates'''<br />
* Ready for 2.2.0 (apart from binary/deps issues)<br />
** Missing some libs (scikit-learn, tej, tdparser, SQLAlchemy+connectors<br />
** Windows: runvistrails.py is no longer used, so the PATH is wrong<br />
** Windows: pip is broken, but it probably wouldn't work anyway because of permission issues (disable this?)<br />
* Queries, upgrades and getPipeline() usage ([https://github.com/VisTrails/VisTrails/issues/1054 #1054])<br />
** Getting a pipeline with getPipeline() is not safe: it might return an invalid pipeline<br />
** This is used in many places throughout the code, like queries<br />
** Upgrading would require going through the controller, but that creates new actions<br />
* [TC] Avoid copying a module's output if it's used as input by exactly one downstream module ([https://github.com/VisTrails/VisTrails/issues/1060 #1060]) (useful for big numpy arrays you can update in-place)<br />
<br />
=== April 8, 2015 ===<br />
'''Updates'''<br />
* [DK] Merged RR's changes for output modules (1012 and 1013)<br />
** RR will merge remaining changes, then create v2.2 branch!<br />
<br />
'''Items to Discuss'''<br />
* BNL need numpy array to VTK image<br />
** Looks like VTK has some helpers for this<br />
** We will help if issues arise<br />
** Will contribute back to VisTrails package<br />
* Upgrade issue: [https://github.com/VisTrails/VisTrails/issues/1017 #1017]<br />
** Automatic upgrades should happen between versions of provided upgrades<br />
** Our existing upgrades work around this so it doesn't need to be 2.2.0<br />
* Corner-case VTK modules<br />
** No longer need VTKCell input port, so don't interfere with registry and API anymore<br />
** Still work weirdly, people probably shouldn't use them<br />
** But we have lots of clunky modules since we wrap the whole of VTK; some people might rely on this and know how to use them, let's keep them anyway<br />
** Ready for 2.2.0<br />
<br />
=== April 1, 2015 ===<br />
'''Updates'''<br />
* [RR] UV-CDAT: bugfixing for 2.2, long-term plans: implement scripting import/export in VisTrails, port to UV-CDAT<br />
* Possibly, try to move to regular VisTrails to use new features<br />
<br />
'''Items to Discuss'''<br />
* 2.2 release: nothing much is pending anymore, release next week?<br />
** [TE] vtkExporter classes [https://github.com/VisTrails/VisTrails/issues/1032 #1032]<br />
** [RR] Hiding upgrades in version tree might make it (or might be 2.2.1 so we can test it out) [https://github.com/VisTrails/VisTrails/issues/949 #949]<br />
** Output modules changes to go in<br />
* add note to documentation about order of parameters in VTK<br />
* add issue about exporter upgrades if not already there<br />
<br />
=== March 25, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* 2.2 release<br />
** Checklist on Github: https://github.com/VisTrails/VisTrails/wiki/2.2-Release-Checklist<br />
** Review output modules<br />
** RR has a couple more issues to fix<br />
** Ready to go -- sign app package for OSX? [https://github.com/VisTrails/VisTrails/issues/984 #984] We need access to the Apple Membership team<br />
<br />
=== March 18, 2015 ===<br />
'''Updates'''<br />
* [TE] VTK6 works<br />
<br />
'''Items to Discuss'''<br />
* [RR] Release v2.1.5 with backported tabledata?<br />
** MTA example needs updated tabledata (for JoinTables)<br />
** Google Maps package still not available<br />
* [RR] Work torward v2.2.0?<br />
** changes:<br />
*** new persistence<br />
*** API changes<br />
*** output module changes (upgrades?), maintain cells but try to upgrade<br />
*** not wrapping stuff<br />
*** VTK6? yes<br />
*** JobSubmission stuff?<br />
*** relabeling for upgrades [https://github.com/VisTrails/VisTrails/issues/949 #949]<br />
** makes sense, needs the tree view code to be updated, check selection<br />
** See [https://github.com/VisTrails/VisTrails/wiki/2.2-Release-Checklist 2.2 checklist]<br />
* Discussion of [https://github.com/VisTrails/VisTrails/issues/1016 #1016]<br />
** plumbing between outputs and output modes, how to define a mode that works for many outputs without writing for each output?<br />
<br />
=== March 11, 2015 ===<br />
'''Updates'''<br />
* [RR] persistent_archive done; merge? ([https://github.com/VisTrails/VisTrails/pull/755 #755])<br />
** note about the focus events for widgets<br />
** TE be aware of file_archive for future binaries that include persistent_archive<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/998 New VTK package] finished<br />
<br />
'''Items to Discuss'''<br />
* [RR] Unmark UV-CDAT/VisTrails as a fork of VisTrails/VisTrails? (see last week; decision needed)<br />
** RR email to JF about this<br />
** yes; email sent to Github<br />
* [RR] Switching order of output ports ([https://github.com/VisTrails/VisTrails/pull/1006 #1006])<br />
** added port specs are sorted at a separate spot (Module.*_port_specs properties) than those in the registry (which are sorted in the registry), but those two lists are just combined without respect to sort keys<br />
** need to determine whether the two lists should be merged or remain distinct<br />
** should make sure that order of input ports and output ports makes parallel connections for things with same order<br />
** DK suggests breaking backward compatibility here: workflows still run, can fix easily if a problem in an existing package.<br />
* [RR] Question about output modes ([https://github.com/VisTrails/VisTrails/pull/1007 #1007]), how to integrate in API ([https://github.com/VisTrails/VisTrails/pull/24 #24])<br />
** Should ImageFileMode be removed? ("image" is not a mode, "file" is)<br />
** ImageOutput missing?<br />
** Feel free to change how formats works<br />
* [TE] Test suite segfaults on Fedora 17 virtualbox. Install newer version? (Support ended in 2013)<br />
<br />
=== March 4, 2015 ===<br />
'''Updates'''<br />
* [TE] VTK wrapper<br />
** Works on VTK 5.10<br />
** Still need to test VTK 6<br />
** New general wrappers for python functions and classes <br />
<br />
'''Items to Discuss'''<br />
* [RR] Unmark UV-CDAT/VisTrails as a fork of VisTrails/VisTrails?<br />
** When [http://i.imgur.com/UTY2qfg.png filing a pull request] from UV-CDAT/VisTrails, VisTrails/VisTrails is selected by default<br />
** People keep forgetting to change the default ([https://github.com/VisTrails/VisTrails/pull/956 #956] [https://github.com/VisTrails/VisTrails/pull/968 #968] [https://github.com/VisTrails/VisTrails/pull/999 #999] [https://github.com/VisTrails/VisTrails/pull/1000 #1000] [https://github.com/VisTrails/VisTrails/pull/1003 #1003] [https://github.com/VisTrails/VisTrails/pull/1004 #1004] [https://github.com/VisTrails/VisTrails/pull/1005 #1005])<br />
** Only way to change that is to not have it marked as a fork of VisTrails/VisTrails<br />
** Github staff [https://gist.github.com/remram44/b364c24ef1e462ce6e2f can make that change for us]; should we do it?<br />
** Juliana: comments on this? visibility vs. convenience/annoyance for developers<br />
* [RR] What about [https://github.com/VisTrails/VisTrails/issues/991 #991]?<br />
<br />
=== February 25, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] vtkviewcell for infovis support, can we unify with VTKCell?<br />
** need to test this<br />
* [TE] vtk wrapping<br />
** Mostly finished<br />
** VTK 5.10 produce incorrect results with old wrapping<br />
*** Old wrapper is based mostly on VTK 4<br />
*** Most vtk_examples affected<br />
*** [http://www.vtk.org/Wiki/VTK/VTK_6_Migration/Replacement_of_SetInput]<br />
*** should be able to upgrade from SetInput to SetInputData (need to drop GetOutput and replace with self ports)<br />
*** can we change vtkInstance to just return self and not wrap things<br />
** terminator example not working under 5.8?<br />
* How does VTK wrapping fit into general wrapping framework?<br />
* [RR] new persistence package<br />
<br />
=== February 18, 2015 ===<br />
'''Updates'''<br />
* [RR] New VisTrails API and IPython integration ([https://github.com/VisTrails/VisTrails/pull/24 #24])<br />
<br />
'''Items to Discuss'''<br />
* [TE] VTK wrapping<br />
** [https://gist.github.com/rexissimus/16b74693545b5e7c1035 Benchmarking vtk package]<br />
*** Old: 24.7 seconds <br />
*** New: 10.5 seconds (Except first time that adds 8 sec)<br />
**** The parsing that calls is_abstract (that tries to instanciate all vtk classes) is now only run the first time.<br />
**** get_items_from_sigstring takes 2 seconds, maybe we can use a lookup dict for already computed sigstrings?<br />
** Now using a general python function wrapper<br />
*** VTK classes are wrapped into python function that does not depend on vistrails<br />
*** VTK functions can be executed without vistrails<br />
*** The spec maps functions into vistrails modules, but can also describe wrapping <br />
*** A general python function wrapper that supports <br />
**** kwarg inputs<br />
**** single, list, dict outputs<br />
**** callback for progress reporting<br />
**** temporary file generator for using FilePool<br />
**** optional output generation<br />
*** Creating specs:<br />
***** Create spec by hand<br />
***** Auto-create spec outline (TODO) and manually finish it<br />
***** Dynamically create spec (VTK)<br />
***** Implement documentation wrappers (Can use scikitlearn wrapper to wrap numpydoc) (TODO)<br />
***** Classes as bad functions needs to be wrapped in new functions before they are wrapped. This is different for each package.<br />
****** Classes is hard: Like VTK, and matplotlib. Scikit-learn does still not wrap classes<br />
***** Spec diffing and patching could be done using code from matplotlib.<br />
** Still needs upgrades from old VTK package<br />
*** Is it possible to dynamically wrap functions, e.g, you see a SetFunc and just remove the 'Set' prefix. Or do you need to create a complete mapping?<br />
<br />
=== February 11, 2015 ===<br />
'''Updates'''<br />
* Update from Friday's meeting<br />
** discussed VisTrails internals<br />
** discussed wrapping<br />
*** xml discussion, hard to modify because tied to db code<br />
*** TE has made it possible to add the schema-defined attributes to the intermediate representation<br />
*** higher-level operations on the port specs<br />
** make sure the simple case works<br />
*** [JF] take a simple package with documentation and figure out what the base case for wrapping is<br />
<br />
'''Items to Discuss'''<br />
* [TE] VTK wrapping<br />
** Dynamic loading works<br />
** Reading XML is fast enough, but [https://gist.github.com/rexissimus/8272d3c776ccfa115d82 unserializing data is slow]<br />
** Working on patterns for patching<br />
** Matplotlib has many advanced patterns like argument ordering, nested arguments, alternateSpecs, output types.<br />
*** Having all this in a general wrapper might confuse users?<br />
** [RR] Delay module (except for identifiers) until you need it---e.g. don't deal with port specs, etc. until necessary<br />
* Scripting Support [https://github.com/VisTrails/VisTrails/pull/950 #950]<br />
** [RR] Issue with getting code from modules<br />
** Design a simple solution<br />
** [JF] Couldn't you use modules as black boxes without conversion, just to call into modules/subworkflows easily from e.g. IPython?<br />
*** [RR] This is a job for the API, and a very separate use case. See [https://github.com/VisTrails/VisTrails/pull/24 #24]<br />
<br />
=== February 4, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* Wrapping<br />
** Format to use? Currently XML (like current matplotlib)<br />
*** JSON and YAML have simple "to python dictionary" methods<br />
*** But don't stream<br />
*** YAML a lot easier for humans<br />
** [DK] vtk-new-package also changes parameter names, creates enumerations<br />
*** intermediate schema needs to be extensible<br />
*** packages will want to store there specific infos for compute() method generation<br />
*** also might have specs-altering info, like matplotlib's alternateSpec<br />
** representation to code , registry already has schema for some aspects<br />
** [RR] We might want to see if Module subclasses can be created lazily<br />
*** no need to create all the classes just to register them in the registry and never actually use most of them<br />
*** future effort<br />
* [RR] Where should VisTrails packages live?<br />
** tej installs as 'vistrailspkg.tej', TE installed it as 'userpackages.tej'<br />
** Currently, standard packages are 'vistrails.packages.', user packages are 'userpackages.' and packages loaded through pkg_resources might be anything<br />
** [RR] Use 'vistrailspkg.' everywhere?<br />
** Long-term effort to simplify package distribution/installation (and have VisTrails get them automatically?)<br />
<br />
=== January 28, 2015 ===<br />
'''Updates'''<br />
* T. Caswell to come visit on Fri 6 to discuss wrapping work<br />
<br />
'''Items to Discuss'''<br />
* [TE] New VTK wrapping<br />
** Current code by DK seems a good deal faster<br />
** Generates XML that can be patched/tweaked, generates Python code from it, VisTrails only loads generated Python code<br />
** RR would rather have VisTrails load intermediate representation (= XML) directly, wants to make sure this is not slower<br />
** The goal is to turn the intermediate step into something generic that would be used for every wrapped package (vtk, numpy, matplotlib, sklearn, java) instead of each having its own<br />
** TC has its own code at [https://github.com/Nikea/VTTools/blob/master/vttools/wrap_lib.py github:VTTools] which parses numpy docstrings and generates modules, doesn't yet handle classes or persist anything<br />
* Web crawler<br />
** Right now, TE starts jobs for "start crawler", "stop crawler", "install classifier"<br />
** RR would rather have the crawling be a job as far as VisTrails and tej are concerned<br />
** The whole thing would be one pipeline: load examples, train classifier, start crawler [check for job, kill previous one, upload model, start processes], get snapshot, visualize<br />
*** Need some support in tej and job submission system: long-running jobs, stop a job (wait for it to finish?), restart a job even though results are cached<br />
<br />
=== January 21, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Unified wrapping method discussion [https://github.com/VisTrails/VisTrails/issues/991 #991]<br />
** TE to work on reusable method with intermediate representation, starting with VTK<br />
* [RR] Examples for scikit-learn: JF has an old example using Weka with parameter exploration (not currently in source tree)<br />
** AM's examples are enough<br />
* [AM] scikit-learn package is done, merge it in? [https://github.com/VisTrails/VisTrails/pull/955#issuecomment-70286788 #955]<br />
** RR will merge<br />
* [RR] What should copyright headers say? [https://github.com/VisTrails/VisTrails/pull/994#issuecomment-70173935 #994]<br />
** Let's keep everything in there: Utah/Poly/NYU<br />
<br />
=== January 14, 2015 ===<br />
'''Updates'''<br />
* [TE] Working on classifier<br />
* [RR] Scripting integration, work in progress<br />
<br />
'''Items to Discuss'''<br />
* [RR] Unified wrapping method discussion ([https://github.com/VisTrails/VisTrails/issues/991 #991])<br />
** Let's talk next week, [AM] and [DK] are not here<br />
<br />
=== January 7, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* make sure that we address critical issues, questions, and pending review branches in a timely manner<br />
* scripting support<br />
** [RR] no issues if we want to just keep annotations in the generated code to allow the link back to a workflow<br />
** [RR] can translate from workflow to script, working on script to workflow<br />
** will work for parameter value changes, structural changes require changes to the annotations<br />
** need to publish best practices here<br />
** would be cool to do looping in scripts (easier interface than with workflows)<br />
* notebook support (convert form notebook to workflow)<br />
** RR will sync with FC on this<br />
* Issue with console in built-from-scratch<br />
** [TC] iPython rearranged some of the completion stuff in 2.2 and 2.3<br />
** binary has old version of iPython -> 1.0.0, should we update?<br />
* [TC] automated wrapping of numpy and scipy<br />
** discovered a bunch of malformed documentation in numpy and scipy<br />
** has github repo for vistrails tools<br />
** example modules wrap a bunch of R stuff (not baked in, just how things are)<br />
** will be pushing wrapping logic up<br />
** port names forbidden (window and domain)<br />
** have an import hook to get from yaml directly to VisTrails Modules<br />
** should work for any python modules with well-formed numpy docstrings.<br />
* [Action] should make it clear in documentation that Constant now means serializable not that the value doesn't change (e.g. List)<br />
* [TC] might be interesting to try to build components of matplotlib and accumulate in figure (long-term project, but thinking about how this might work)<br />
* [TE] build and build scripts<br />
** completely automatic, buildbot<br />
** need to set the build machines for the environment we want for the binary<br />
** would virtualenv work here?<br />
** [TC] anaconda can pin versions, potential path to test different configurations<br />
** Q: upload nightly binary builds? A: makes sense, make sure they are well-labeled<br />
* sourceforge stats: e.g. http://sourceforge.net/projects/vistrails/files/vistrails/nightly/vistrails-src-nightly.tar.gz/stats/timeline?dates=2014-01-01+to+2015-01-07<br />
* package issues (see Remi's message)<br />
* [TE] Scope of tej<br />
** Support single ssh commands?<br />
** Queue can be used as a remote machine (crawler is using queue.call*)<br />
* SourceForge stats: http://sourceforge.net/projects/vistrails/files/vistrails/nightly/vistrails-src-nightly.tar.gz/stats/timeline?dates=2014-01-01+to+2015-01-07<br />
<br />
== Older meetings ==<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=11864Development2016-05-25T15:58:42Z<p>Remi: /* May 25, 2016 */</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2016 ==<br />
<br />
=== May 25, 2016 ===<br />
<br />
'''Updates'''<br />
* Mailing-list fixed<br />
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])<br />
<br />
'''Items to Discuss'''<br />
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)<br />
* [TE] Union ports<br />
** Show union ports in pipeline (DEMO)<br />
<br />
=== May 18, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] mailing list still down - Julio working on it<br />
* [TE] Improved union port<br />
** New schema 1.0.5 adds portitem.union string attribute<br />
** Works with pipeline view, parameter exploration view, and mashup view<br />
** Should work well for matplotlib<br />
** Use with VTK ports?<br />
*** It uses [item_1,item_2] style ports usually with different tuple length.<br />
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]<br />
<br />
=== May 11, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?<br />
*** Shows single port for all ports with same sort_key<br />
*** Select specific type when adding/showing<br />
*** Only one type should be visible on the module at any time<br />
*** Making tuple parameters union types is probably too complicated<br />
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly) <br />
<br />
* [TE] Feature suggestion: Let connections connect to functions/parameters<br />
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters<br />
** Use separate views for ports and functions (like before)<br />
*** Allow rearranging of functions<br />
** Pros:<br />
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.<br />
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.<br />
**** Makes List module simpler<br />
*** Connect to parameter in tuple<br />
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds. <br />
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.<br />
** Cons:<br />
*** Interface more complicated and crowded<br />
*** More difficult for users to grasp?<br />
*** No immediate use case<br />
*** Makes no union type for tuple parameters a bigger problem.<br />
<br />
* [TE] Publish user survey?<br />
** No response from vistrails release yet<br />
** Not announced on vistrails-users yet.<br />
<br />
=== May 4, 2016 ===<br />
'''Updates'''<br />
* VisTrails 2.2.4 released<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
<br />
=== April 27, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Wrapping Bokeh<br />
** New method type "operation"<br />
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output<br />
**** Glyph sets function name and arguments as output<br />
***** Not serializable, but should always be computed together with the subject of the operation.<br />
**** Figure applies Glyph operation on itself<br />
*** Useful for matplotlib?<br />
**** matplotlib currently find actors by searching upstream?<br />
**** But this is more difficult since matplotlib is not function-based.<br />
** New "union" port type<br />
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value<br />
**** Uses only first value in tuple<br />
**** Bokeh have many properties that support (list, string, Float)<br />
*** Does not work with depth=1 types (e.g. Float of depth 1)<br />
**** Turn Depth 1 types with known type into lists?<br />
<br />
* [TE] New bugfix release<br />
** Finalize survey<br />
<br />
=== April 20, 2016 ===<br />
'''Updates'''<br />
* Merged upgrade fixes<br />
** Now has infinite loop test<br />
** loop test now correctly loops 5 times<br />
*** remap_module used wrong (current) package version when creating ModuleDescriptor<br />
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])<br />
<br />
'''Items to Discuss'''<br />
* New bugfix release<br />
** Merge usagestats<br />
** Need to finalize survey<br />
<br />
=== April 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?<br />
<br />
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])<br />
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]<br />
** Cherry-pick test to preserve source?<br />
** Need maxPipelineFixAttempts option?<br />
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.<br />
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?<br />
*** Would need to be moved into handle_invalid_pipeline<br />
<br />
=== April 6, 2016 ===<br />
'''Updates'''<br />
* [TE] Query View improvements<br />
** Done except for displaying "Version Info" and #1164<br />
* [RR] Working on finishing new interpreter (finally!)<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Try latest upgrade first, then previous.<br />
** If all fail, select latest upgrade?<br />
<br />
=== March 30, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Query View improvements<br />
** Unified do_version_switch and get_upgrades as validate_version()<br />
** Problems with query view using its own controller<br />
*** Fixed module info not being displayed for query results<br />
*** Still no version info for query results<br />
*** Still an issue with version query view not being updated<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== 2015 ==<br />
<br />
=== December 30, 2015 ===<br />
'''Updates'''<br />
* Bokeh package finished<br />
** Can output to Browser/IPython/Spreadsheet (Limited functionality in spreadsheet)<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Bokeh needs QWebEngine (Chromium) in Qt5 which requires PyQt5<br />
*** Many plotting libraries only work in browsers<br />
*** But Chromium may be a big and controversial dependency<br />
*** Or create a browser version of the spreadsheet?<br />
** Some users have requested Python 3<br />
** Not backwards compatible?<br />
** Target for VisTrails 3.0 with new interpreter?<br />
<br />
=== December 16, 2015 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Users list question<br />
* [TE] Wrapping Bokeh<br />
** Added bokeh property parser<br />
*** complements numpydoc parser<br />
*** Not all attributes are available, partially because I am using a pre-release version.<br />
** Wrapped modules<br />
*** bokeh.plotting finished (Backwards like matplotlib: Create plot then draw on it)<br />
*** bokeh.charts mostly done (Missing arguments)<br />
** QWebView mostly works (but no zooming or buttons)<br />
*** It would be better to use external browser or QWebEngineView<br />
** Added improvements to wrapper<br />
*** Generalized port translations (Color/Path to native types)<br />
*** Can use multiple docstring/property parsers simultaneously<br />
<br />
=== December 9, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Looking into wrapper<br />
** Will port TensorFlow<br />
** Doing a pass on the wrapper code & doc<br />
*** UX improvements<br />
** Factorize high-level docstring parsers into core.wrapper? (sphinx, numpy, google)<br />
** Problem with name vs module_name (fails sklearn tests because class_by_name() is broken)<br />
** Looking into compressing specs, lazy registration of modules<br />
<br />
=== December 2, 2015 ===<br />
'''Updates'''<br />
* [TE] Package wrapping<br />
** Wrapped most of numpy/scipy using __all__<br />
** Added class attribute and method access for classes (Can add to class or to separate inspector module)<br />
** TODO: Document wrapping procedure<br />
<br />
'''Items to Discuss'''<br />
<br />
=== November 25, 2015 ===<br />
'''Updates'''<br />
* [RR] New interpreter is coming along<br />
** Everything is a stream<br />
** Task system, eventually work-stealing parallelism<br />
** Streams no longer advance in lock-step<br />
*** Allows for constructions like filter, join, sorted-merge<br />
** depth>1 still to be tested, no current plan to allow this through the module interface<br />
*** Through looping though, can do streams of streams<br />
<br />
'''Items to Discuss'''<br />
* [TE] Package wrapping<br />
** Wrapped most numpy functions (no polynomials)<br />
** and half of scipy classes and functions<br />
** Added functions manually from documentation (no parseable lists?)<br />
** Focusing on data flow constructs (no property/method access for classes)<br />
*** Using spec diff (empty for now) and dynamic parser (stores spec in .vistrails/)<br />
** Python class/function wrapper now stable<br />
** Add google doc parser?<br />
*** Is there a parser available?<br />
<br />
=== November 18, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Package wrapping<br />
** Numpy class wrapping works<br />
*** Basic type is List, since most types are array_likes.<br />
*** TODO: More classes and functions<br />
*** Some operations are in-place and some aren't, and docstring not super clear<br />
*** [RR] It seems only methods mutate, and most have a numpy.xx function equivalent; just manually go through the methods and remove the mutating ones?<br />
** Class wrapper is modular<br />
*** Docstring parser and type string parser can be customized<br />
*** Classes can have optional attribute/method ports<br />
**** Inspectors and attribute/method modules can be created separately<br />
** Use PythonCalc as an example<br />
*** Can be re-implemented as a function with a parseable docstring<br />
*** [RR] PythonCalc is just an example, and it might actually make sense to get rid of it, or do a proper math package with scalar operations (as separate modules, no combobox)<br />
** [RR] TensorFlow to use wrapping as well<br />
*** Very simple (only types 'tensor' & 'variable')<br />
<br />
=== November 11, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Package wrapping<br />
** Wrapping numpy's classes using numpydoc (ndarray)<br />
*** Wrapping constructor arguments, attribute getters/setters as module ports<br />
**** Then how to access attributes afterwards?<br />
***** Use input value port to class modules?<br />
*** Wrap methods as modules? E.g., `ndarray.shape`.<br />
*** Function wrapping is almost a subset of class wrapping<br />
**** May be able to use the same parsing/execution methods<br />
** VTK's non-getter/setter methods could be wrapped as modules<br />
*** We could then remove extra logic in interpreter for keeping function order<br />
** Bokeh uses autogenerated docstrings<br />
*** We may be able to read the specification directly<br />
<br />
* [RR] [https://github.com/VisTrails/VisTrails/pull/1138 TensorFlow package]<br />
** Basic setup working, can execute the Mandelbrot example<br />
** Will autogenerate the operations<br />
<br />
=== November 4, 2015 ===<br />
'''Updates'''<br />
* [RR] Interpreter work requires a fix for [https://github.com/VisTrails/VisTrails/issues/1137 DB issue #1137]<br />
* [DK] Kitware's Resonant [http://www.kitware.com/source/home/post/173]<br />
** [https://github.com/girder/girder Girder]: data management system<br />
** [https://github.com/Kitware/romanesco Romanesco]: execution engine, uses [http://www.celeryproject.org/ Celery] for task management<br />
** Resonant Flow: web app for editing and executing workflows<br />
<br />
'''Items to Discuss'''<br />
* [TE] Added OSX Lion (10.7) VM on build machine using vagrant (seems ok with license?)<br />
** New builds works on Lion<br />
** Change the minimum requirement to 10.7?<br />
* [TE] Package wrapping<br />
* Added upgrade suggestions using 2 spec versions ([http://pastebin.com/30XUU8Gp example])<br />
** using name edit distance to find matches<br />
** Could use something better like port similarity for modules, and type similarity for ports<br />
** Should write python upgrade code?<br />
** Added spec to sklearn package<br />
*** Can now diff spec versions and keep the spec static<br />
*** There will be problems with downgrades and package versions<br />
<br />
=== October 29, 2015 ===<br />
'''Updates'''<br />
* [TE] nightly-build-mac fixed (Needed to approve xcode license)<br />
<br />
'''Items to Discuss'''<br />
* [TE] matplotlib wrapper<br />
** ported to general spec<br />
** Diff can be used on general spec<br />
*** Only needed minor changes<br />
*** Fixed indexing bug that corrupted specs<br />
*** Can now move diff tools to core/wrapper<br />
** Add new plots?<br />
*** New plots in mpl 1.3: eventplot<br />
*** New plots in mpl 1.4: angle_spectrum, magnitude_spectrum, phase_spectrum', violinplot<br />
*** No new plots in mpl 1.5 (from looking at boilerplate.py)<br />
<br />
=== October 21, 2015 ===<br />
'''Updates'''<br />
* [RR] Alexis has arrived, will be working with [RR] on a more efficient interpreter<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1133 New matplotlib package]<br />
** Supporting multiple package versions<br />
*** Check which version can be loaded [https://github.com/VisTrails/VisTrails/pull/1135 #1135]<br />
*** We cannot show version requirements in the list of packages because old packages are loaded when importing the codepath<br />
** Visual diff does not work well, but we can use the spec differ to see differences.<br />
** Unifying vtk and matplotlib wrapping specs (WIP)<br />
*** And porting matplotlibs spec differ to work on the general spec<br />
*** Spec differ can then be used on vtk and other packages in the future <br />
*** Will enable automatic upgrade generation<br />
* [RR] Rework interpreter<br />
** restore abstraction between interpreter/module code<br />
** build looping, streaming, caching into the interpreter<br />
** rework cache<br />
** look into parallelism once basic functionality is there<br />
<br />
=== October 14, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1133 New matplotlib package] (adding numpydoc parser)<br />
<br />
=== October 7, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://github.com/VisTrails/VisTrails/tree/fix-matplotlib-parser New matplotlib parser] (adding numpydoc parser)<br />
** Added numpydoc attribute parser for plots.<br />
*** TODO: Need better port spec reconciliation with call signature parser.<br />
*** We can create a general numpydoc parser, but attribute types are unique to matplotlib.<br />
** Needs package versioning<br />
*** New spec will not support matplotlib < v1.4 due to changed path to axes classes.<br />
*** Load spec version corresponding to installed matplotlib version?<br />
*** Need version downgrades?<br />
<br />
=== September 30, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Updating Matplotlib parser<br />
** Matplotlib docstring parser fails on numpy docstrings<br />
** I have [https://github.com/VisTrails/VisTrails/commit/eae8b0c16260810587e47c41cab53552cbb3cf0a added basic numpydoc parsing]<br />
*** Only used by a few docstrings so far<br />
*** Parsing uses many sources (class tables, signatures, ACCEPTS, method docstrings, definition parsing).<br />
** Caswell said they were thinking about moving to traitlets, but this is not ready yet<br />
<br />
=== September 23, 2015 ===<br />
'''Updates'''<br />
* [TE] VisTrails 2.2.3 released<br />
** Also have pushed to PyPI, binstar, etc.<br />
** Sourceforge vs. GitHub<br />
*** should be able to host releases on GitHub<br />
*** nightly binaries pushed to sf each night<br />
*** old binaries?<br />
* [TE] Re-wrapping MatPlotlib<br />
** Keep static generation - Docstrings are brittle beween matplotlib versions<br />
** Keep generating the executable classes - but create functions, not vistrails Modules <br />
*** Why is there so much patching?<br />
** Re-implemented Alternate PortSpec for InputPortSpec<br />
*** It will now inherit specs from parent (No need to reimplement)<br />
** Module upgrades?<br />
<br />
'''Items to Discuss'''<br />
<br />
=== September 16, 2015 ===<br />
'''Updates'''<br />
* [TE] Job Monitor tests and documentation done<br />
** OK to leave document package in packages directory?<br />
<br />
'''Items to Discuss'''<br />
* [TE] Library wrapping: How to do code patching? [https://docs.google.com/drawings/d/1Sh3eOXot6v2y880jHKQrqXqAwr-0aoqYligg3z8G8MY/edit?usp=sharing Wrapping diagram]<br />
** Insert code into module template (Old VTK code)<br />
** Patch the library that is called (New VTK code)<br />
** Somehow store code in spec and apply when executed?<br />
*** Executed code needs at a minimum access to inputs, outputs, current class<br />
*** Use standard input/output dict and obj reference that the code operates on?<br />
*** Can this be done while keeping the execution abstract?<br />
<br />
* Do new release now?<br />
<br />
=== September 9, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* New release?<br />
** [RR] wants reprounzip<br />
** tej changes & doc<br />
** job monitor & jobmixin fixes<br />
** mongodb<br />
** reprounzip<br />
** warning: MplFigure type is int<br />
** tabledata: add Reshape, DictoToTable & ListToTable, fix ListToTable with numpy arrays<br />
** PythonSource can have same name for an input & output<br />
** don't show spreadsheet at exit<br />
<br />
=== September 2, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Limit autosaves? [https://github.com/VisTrails/VisTrails/pull/1126 #1126]<br />
<br />
=== August 26, 2015 ===<br />
'''Updates'''<br />
* [TE] Added Job support to Parameter Explorations (Requested by Colin), and Mashups<br />
** specify job ids, need to specify different ids for parameter explorations since they have the same version id<br />
** how to deal with parameters passed in on command line<br />
* [General] provenance: should be creating a new version when we execute workflow with changed parameters?<br />
** currently, custom_params annotation stores this in provenance currently<br />
** [RR] API doesn't record provenance if passing in parameters: http://git.io/vsAA5<br />
'''Items to Discuss'''<br />
<br />
=== August 19, 2015 ===<br />
'''Updates'''<br />
* [RR] Improvements to tej, users' guide entry & example [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
* [RR] Internal docs [https://github.com/VisTrails/VisTrails/pull/1116 #1116]<br />
** On ReadTheDocs: [http://vistrails.readthedocs.org/ vistrails.readthedocs.org]<br />
* [TE] Working on Job Monitor documentation<br />
<br />
'''Items to Discuss'''<br />
* [RR] JobMixin and JobMonitor: stable now?<br />
** Definitely needs more tests<br />
* [TE] Stop testing VisTrails 2.0?<br />
<br />
=== August 12, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Internal docs [https://github.com/VisTrails/VisTrails/pull/1116 #1116]<br />
** On ReadTheDocs: [http://vistrails.readthedocs.org/ vistrails.readthedocs.org]<br />
* [RR] JobMixin and JobMonitor: stable now?<br />
** Definitely needs more tests<br />
* [RR] Improvements to tej, users' guide entry & example [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
* [RR] Build broken on Travis; because of IPython 4 released today? ([https://travis-ci.org/VisTrails/VisTrails/builds/75279795 build 992]; [https://github.com/VisTrails/VisTrails/issues/1123 #1123])<br />
<br />
=== August 5, 2015 ===<br />
'''Updates'''<br />
* [TE] Fixes to Jobs<br />
** Could not delete jobs<br />
** Could not run job in group<br />
** Job not reset when calling ModuleSuspended<br />
** Added deleting job from context menu<br />
* [RR] Writing documentation for everything [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
<br />
'''Items to Discuss'''<br />
<br />
=== July 29, 2015 ===<br />
'''Updates'''<br />
* RR still looking into new interpreter thing<br />
** Goal is to take out scheduling logic from Module so it can be split in multiple processes, and so that smarter strategies can be added in time<br />
** This means some work on packages<br />
** Spreadsheet can live in kernel process? Still some UI stuff that will take work (changing configuration, persistent archive's viewer, ...)<br />
* [TE] Problems running examples [https://github.com/VisTrails/VisTrails/pull/1111 #1111]<br />
** Testing of more examples requires additional packages on the test machines.<br />
** Fixed faulty line-ending in PythonSource:s failing on Python 2.6.<br />
** Test suite now testing SUDSWebServices (If web service is down, test suite will fail)<br />
** preferences.py test failed reloading 'dialogs' package, switched to using 'URL'.<br />
<br />
'''Items to Discuss'''<br />
<br />
=== July 22, 2015 ===<br />
'''Updates'''<br />
* [TE] mailing lists back online<br />
* [TE] Working on [https://github.com/VisTrails/VisTrails/issues/1107 #1107]<br />
* [RR] Working on ReproZip package <br />
<br />
'''Items to Discuss'''<br />
* [RR] MongoDB package [https://github.com/VisTrails/VisTrails/pull/1106 #1106]<br />
* [RR] Example for tej docs? [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
<br />
=== July 15, 2015 ===<br />
'''Updates'''<br />
* 2.2.2 released<br />
<br />
'''Items to Discuss'''<br />
* RR is considering executing everything in an IPython kernel (i.e. separate Python interpreter, like the one spawned for a notebook)<br />
** It's a separate process that we can restart without restarting the app/gui<br />
** We can isolate the execution environment (e.g. for the server)<br />
** We can run the whole thing remotely (if your data is elsewhere, just run VisTrails locally and the pipeline elsewhere)<br />
** Pipeline execution no longer makes the interface hang, it just makes the kernel hang (but that's fine)<br />
** We can use notebooks as modules (probably way nicer than the PythonSource module)<br />
** We can run multiple kernels so long as the ports carry things that are serializable<br />
*** meaning we can put the multithreaded-interpreter without all the hacky parts it has now<br />
*** we can run IPython kernels in all the languages IPython supports, [https://github.com/ipython/ipython/wiki/IPython-kernels-for-other-languages currently 46]<br />
<br />
=== July 8, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Buildbot github hook not working after IP address change<br />
* [TE] No reference to VistrailsApplication ([https://github.com/VisTrails/VisTrails/issues/1103 #1103])<br />
** [RR] Added PR [https://github.com/VisTrails/VisTrails/pull/1104 #1104]<br />
* [TE] Reopening VT file after saving with bundled subworkflow won't offer subworkflow upgrade ([https://github.com/VisTrails/VisTrails/issues/1102 #1102])<br />
** allow manual delete to fix right now<br />
** fix this on top of the use-uuid branch<br />
* [TE] Release VisTrails 2.2.1? ([https://github.com/VisTrails/VisTrails/blob/v2.2/CHANGELOG CHANGELOG])<br />
** [https://github.com/VisTrails/VisTrails/pull/1093 output-modules-use-global-dir]<br />
** [https://github.com/VisTrails/VisTrails/milestones/version%202.2 Current issues for 2.2]<br />
<br />
=== July 1, 2015 ===<br />
'''Updates'''<br />
* [TE] PROV fixed<br />
* [TE] Working on subworkflow issues<br />
* [RR] Considering reworking the controller (log vs exception problem, retained upgrades causing interferences) and also the interpreter (IPython?)<br />
<br />
'''Items to Discuss'''<br />
<br />
=== June 24, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* UV-CDAT<br />
* [TE] Can a cyclic workflow be valid? ([https://github.com/VisTrails/VisTrails/issues/1097 #1097])<br />
** focus on disabling the ability to create cyclic pipelines because more things break than just this with a cyclic pipeline<br />
* [RR] Relative paths ([https://github.com/VisTrails/VisTrails/pull/1057 #1057])<br />
** This interacts with the new bundle; how do handle packing files inside the VT bundle?<br />
<br />
=== June 17, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] current_version and reusing existing upgrades are broken ([https://github.com/VisTrails/VisTrails/issues/1095 ticket #1095])<br />
** Could be that export to PROV is using an unflushed upgrade pipeline?<br />
** current_version would then be correct.<br />
** It may work to flush the actions before exporting?<br />
<br />
=== June 10, 2015 ===<br />
'''Updates'''<br />
* DAT: fixed VTK issue on Linux and Mac<br />
* Still crashes on Windows. Need help! Reminder: this works in the VisTrails spreadsheet (QCellPresenter), although no widget get changed there during a drag<br />
** Is it a VTK bug?<br />
** Is it simply impossible to change widgets during the drag, should we do it a different way?<br />
** Did I miss something that is done in VisTrails but somehow not in DAT?<br />
** Low prio, UV-CDAT doesn't run on Windows anyway<br />
<br />
'''Items to Discuss'''<br />
* [TE] Executable Paper ([https://github.com/VisTrails/VisTrails/issues/1086 ticket #1088] [https://github.com/VisTrails/VisTrails/pull/1088 Updated pull request])<br />
* Single instance code and batch mode<br />
<br />
=== June 3, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Executable Paper ([https://github.com/VisTrails/VisTrails/issues/1086 ticket #1088] [https://github.com/VisTrails/VisTrails/pull/1088 pr])<br />
** Requires fixes to command line parameters, Output modules, and batch mode<br />
** How to test this<br />
** Updated missing/outdated flags<br />
** Fixed view issues when generating graphs<br />
* [TE] batch mode<br />
** SpreadsheetOutput not enabled in batch mode. Should we check is_running_gui instead?<br />
** Other instance setting flags from caller<br />
** Is graphsAsPdf replacing spreadsheetDumpPdf?<br />
** graphsAsPdf true by Default?<br />
** Batch mode executing by default (Not needed when generating graphs)<br />
** Re-added workflowInfo as withWorkflowInfo for writing graph and xml workflow<br />
** [DK] batch mode should be outputting to files or stdout, shouldn't always trigger SpreadsheetMode<br />
** execute flag, maybe make execution the default and allow a "--no-execute" if you only want to capture graphs, for example<br />
<br />
=== May 27, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] API question<br />
** http://lists.vistrails.org/pipermail/vistrails-users/2015-May/000357.html<br />
* Status of DAT<br />
** Can add VCS plot to DAT but need to work on configuration windows<br />
** Documentation for graphics templates for UV-CDAT/vcs?<br />
* Qt support?<br />
<br />
=== May 20, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Reviving the DAT, integrating scripting & porting UV-CDAT!<br />
** On [https://github.com/VisTrails/DAT GitHub] ([https://github.com/VisTrails/DAT/issues issues])<br />
** Merging 2 years of development taking longer than expected, but getting there. The plan is to get the patches in VisTrails and never fork again, we never want to get in UV-CDAT's situation (and don't need to).<br />
** VTK cell works fine on Linux but there was flickering on Mac & Windows before; still issues on Mac (Windows status unknown)<br />
** Can get a VCS plot soon (but will need VTK cell fix)<br />
** Integrate in UV-CDAT's build system (so we have cdms, VCS, ...) -> RR can do this, low priority<br />
** How do we integrate scripting?<br />
*** We want to be able to seamlessly make changes to a plot by changing Python code<br />
*** Define new plots by entering Python code without writing modules/packages?<br />
* [RR] UV-CDAT [https://github.com/UV-CDAT/uvcdat/issues/1275 needs] [https://github.com/VisTrails/VisTrails/pull/1073 #1073], please take a look<br />
* [RR] Subworkflow issues<br />
** [https://github.com/VisTrails/VisTrails/issues/1065 #1065]: missing attribute (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/1066 #1066]: groups in subworkflows were ignored when finding dependencies (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/725 #725]: "upgrade subworkflow" button not popping up (fixed); also, namespace issue<br />
** [https://github.com/VisTrails/VisTrails/issues/1071 #1071]: subworkflows don't go through upgrades<br />
* [RR] [https://github.com/VisTrails/VisTrails/issues/1074 #1074]: can't load and edit a single pipeline if upgrade happen<br />
<br />
=== May 13, 2015 ===<br />
'''Updates'''<br />
* [RR] Test skipping whitelist ([https://github.com/VisTrails/VisTrails/pull/1069 #1069]) -- low priority<br />
* [RR] Custom matplotlib modules can't be compatible with both 2.1 and 2.2 ([https://github.com/VisTrails/VisTrails/issues/1067 #1067]); should be fixed for ALPS ([https://github.com/VisTrails/VisTrails/issues/1070 #1070])<br />
<br />
'''Items to Discuss'''<br />
* [RR] UV-CDAT [https://github.com/UV-CDAT/uvcdat/issues/1275 needs] [https://github.com/VisTrails/VisTrails/pull/1073 #1073], please take a look<br />
* [RR] Subworkflow issues<br />
** [https://github.com/VisTrails/VisTrails/issues/1065 #1065]: missing attribute (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/1066 #1066]: groups in subworkflows were ignored when finding dependencies (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/725 #725]: "upgrade subworkflow" button not popping up (fixed); also, namespace issue<br />
** [https://github.com/VisTrails/VisTrails/issues/1071 #1071]: subworkflows don't go through upgrades<br />
* [RR] [https://github.com/VisTrails/VisTrails/issues/1074 #1074]: can't load and edit a single pipeline if upgrade happen<br />
<br />
=== May 7, 2015 ===<br />
'''Updates'''<br />
* [RR] Export/import workflow to Python working!<br />
<br />
'''Items to Discuss'''<br />
* [Claudio] UV-CDAT<br />
** The UV-CDAT project is the biggest user base of VisTrails<br />
** VisTrails package management provides a lot of friction towards people plugging in their code<br />
** Need to make it easy to integrate your random Python scripts in the system without having to deal with all the boilerplate, at least in the first step<br />
** [RR] argues that modules are still good; UV-CDAT shouldn't move towards a purely script-based backend<br />
** [RR] export/import with Python could reduce a lot of that friction by allowing 1) to edit workflow as Python 2) to open up boxes automatically if needed code doesn't match actual modules<br />
** ...<br />
<br />
=== April 29, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] <strike>Abstractions</strike> subworkflows status ([https://github.com/VisTrails/VisTrails/labels/subworkflows tickets])<br />
* [RR] matplotlib compatibility (2.1 & 2.2), [https://github.com/VisTrails/VisTrails/issues/1067 #1067]<br />
** RR to try and fix ALPS matplotlib modules<br />
<br />
=== April 22, 2015 ===<br />
'''Updates'''<br />
* 2.2.0 has been released!<br />
* Windows issue (via email from Ryan)<br />
** issue with manifest file (may be a new file in VTK6?)<br />
** Tommy has regenerated new Windows builds<br />
* Binaries, pypi, and conda released<br />
* [RR] Export as script<br />
* Python sources using VTK need to switch to SetInputData (users should be aware of this)<br />
<br />
=== April 15, 2015 ===<br />
'''Updates'''<br />
* Ready for 2.2.0 (apart from binary/deps issues)<br />
** Missing some libs (scikit-learn, tej, tdparser, SQLAlchemy+connectors<br />
** Windows: runvistrails.py is no longer used, so the PATH is wrong<br />
** Windows: pip is broken, but it probably wouldn't work anyway because of permission issues (disable this?)<br />
* Queries, upgrades and getPipeline() usage ([https://github.com/VisTrails/VisTrails/issues/1054 #1054])<br />
** Getting a pipeline with getPipeline() is not safe: it might return an invalid pipeline<br />
** This is used in many places throughout the code, like queries<br />
** Upgrading would require going through the controller, but that creates new actions<br />
* [TC] Avoid copying a module's output if it's used as input by exactly one downstream module ([https://github.com/VisTrails/VisTrails/issues/1060 #1060]) (useful for big numpy arrays you can update in-place)<br />
<br />
=== April 8, 2015 ===<br />
'''Updates'''<br />
* [DK] Merged RR's changes for output modules (1012 and 1013)<br />
** RR will merge remaining changes, then create v2.2 branch!<br />
<br />
'''Items to Discuss'''<br />
* BNL need numpy array to VTK image<br />
** Looks like VTK has some helpers for this<br />
** We will help if issues arise<br />
** Will contribute back to VisTrails package<br />
* Upgrade issue: [https://github.com/VisTrails/VisTrails/issues/1017 #1017]<br />
** Automatic upgrades should happen between versions of provided upgrades<br />
** Our existing upgrades work around this so it doesn't need to be 2.2.0<br />
* Corner-case VTK modules<br />
** No longer need VTKCell input port, so don't interfere with registry and API anymore<br />
** Still work weirdly, people probably shouldn't use them<br />
** But we have lots of clunky modules since we wrap the whole of VTK; some people might rely on this and know how to use them, let's keep them anyway<br />
** Ready for 2.2.0<br />
<br />
=== April 1, 2015 ===<br />
'''Updates'''<br />
* [RR] UV-CDAT: bugfixing for 2.2, long-term plans: implement scripting import/export in VisTrails, port to UV-CDAT<br />
* Possibly, try to move to regular VisTrails to use new features<br />
<br />
'''Items to Discuss'''<br />
* 2.2 release: nothing much is pending anymore, release next week?<br />
** [TE] vtkExporter classes [https://github.com/VisTrails/VisTrails/issues/1032 #1032]<br />
** [RR] Hiding upgrades in version tree might make it (or might be 2.2.1 so we can test it out) [https://github.com/VisTrails/VisTrails/issues/949 #949]<br />
** Output modules changes to go in<br />
* add note to documentation about order of parameters in VTK<br />
* add issue about exporter upgrades if not already there<br />
<br />
=== March 25, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* 2.2 release<br />
** Checklist on Github: https://github.com/VisTrails/VisTrails/wiki/2.2-Release-Checklist<br />
** Review output modules<br />
** RR has a couple more issues to fix<br />
** Ready to go -- sign app package for OSX? [https://github.com/VisTrails/VisTrails/issues/984 #984] We need access to the Apple Membership team<br />
<br />
=== March 18, 2015 ===<br />
'''Updates'''<br />
* [TE] VTK6 works<br />
<br />
'''Items to Discuss'''<br />
* [RR] Release v2.1.5 with backported tabledata?<br />
** MTA example needs updated tabledata (for JoinTables)<br />
** Google Maps package still not available<br />
* [RR] Work torward v2.2.0?<br />
** changes:<br />
*** new persistence<br />
*** API changes<br />
*** output module changes (upgrades?), maintain cells but try to upgrade<br />
*** not wrapping stuff<br />
*** VTK6? yes<br />
*** JobSubmission stuff?<br />
*** relabeling for upgrades [https://github.com/VisTrails/VisTrails/issues/949 #949]<br />
** makes sense, needs the tree view code to be updated, check selection<br />
** See [https://github.com/VisTrails/VisTrails/wiki/2.2-Release-Checklist 2.2 checklist]<br />
* Discussion of [https://github.com/VisTrails/VisTrails/issues/1016 #1016]<br />
** plumbing between outputs and output modes, how to define a mode that works for many outputs without writing for each output?<br />
<br />
=== March 11, 2015 ===<br />
'''Updates'''<br />
* [RR] persistent_archive done; merge? ([https://github.com/VisTrails/VisTrails/pull/755 #755])<br />
** note about the focus events for widgets<br />
** TE be aware of file_archive for future binaries that include persistent_archive<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/998 New VTK package] finished<br />
<br />
'''Items to Discuss'''<br />
* [RR] Unmark UV-CDAT/VisTrails as a fork of VisTrails/VisTrails? (see last week; decision needed)<br />
** RR email to JF about this<br />
** yes; email sent to Github<br />
* [RR] Switching order of output ports ([https://github.com/VisTrails/VisTrails/pull/1006 #1006])<br />
** added port specs are sorted at a separate spot (Module.*_port_specs properties) than those in the registry (which are sorted in the registry), but those two lists are just combined without respect to sort keys<br />
** need to determine whether the two lists should be merged or remain distinct<br />
** should make sure that order of input ports and output ports makes parallel connections for things with same order<br />
** DK suggests breaking backward compatibility here: workflows still run, can fix easily if a problem in an existing package.<br />
* [RR] Question about output modes ([https://github.com/VisTrails/VisTrails/pull/1007 #1007]), how to integrate in API ([https://github.com/VisTrails/VisTrails/pull/24 #24])<br />
** Should ImageFileMode be removed? ("image" is not a mode, "file" is)<br />
** ImageOutput missing?<br />
** Feel free to change how formats works<br />
* [TE] Test suite segfaults on Fedora 17 virtualbox. Install newer version? (Support ended in 2013)<br />
<br />
=== March 4, 2015 ===<br />
'''Updates'''<br />
* [TE] VTK wrapper<br />
** Works on VTK 5.10<br />
** Still need to test VTK 6<br />
** New general wrappers for python functions and classes <br />
<br />
'''Items to Discuss'''<br />
* [RR] Unmark UV-CDAT/VisTrails as a fork of VisTrails/VisTrails?<br />
** When [http://i.imgur.com/UTY2qfg.png filing a pull request] from UV-CDAT/VisTrails, VisTrails/VisTrails is selected by default<br />
** People keep forgetting to change the default ([https://github.com/VisTrails/VisTrails/pull/956 #956] [https://github.com/VisTrails/VisTrails/pull/968 #968] [https://github.com/VisTrails/VisTrails/pull/999 #999] [https://github.com/VisTrails/VisTrails/pull/1000 #1000] [https://github.com/VisTrails/VisTrails/pull/1003 #1003] [https://github.com/VisTrails/VisTrails/pull/1004 #1004] [https://github.com/VisTrails/VisTrails/pull/1005 #1005])<br />
** Only way to change that is to not have it marked as a fork of VisTrails/VisTrails<br />
** Github staff [https://gist.github.com/remram44/b364c24ef1e462ce6e2f can make that change for us]; should we do it?<br />
** Juliana: comments on this? visibility vs. convenience/annoyance for developers<br />
* [RR] What about [https://github.com/VisTrails/VisTrails/issues/991 #991]?<br />
<br />
=== February 25, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] vtkviewcell for infovis support, can we unify with VTKCell?<br />
** need to test this<br />
* [TE] vtk wrapping<br />
** Mostly finished<br />
** VTK 5.10 produce incorrect results with old wrapping<br />
*** Old wrapper is based mostly on VTK 4<br />
*** Most vtk_examples affected<br />
*** [http://www.vtk.org/Wiki/VTK/VTK_6_Migration/Replacement_of_SetInput]<br />
*** should be able to upgrade from SetInput to SetInputData (need to drop GetOutput and replace with self ports)<br />
*** can we change vtkInstance to just return self and not wrap things<br />
** terminator example not working under 5.8?<br />
* How does VTK wrapping fit into general wrapping framework?<br />
* [RR] new persistence package<br />
<br />
=== February 18, 2015 ===<br />
'''Updates'''<br />
* [RR] New VisTrails API and IPython integration ([https://github.com/VisTrails/VisTrails/pull/24 #24])<br />
<br />
'''Items to Discuss'''<br />
* [TE] VTK wrapping<br />
** [https://gist.github.com/rexissimus/16b74693545b5e7c1035 Benchmarking vtk package]<br />
*** Old: 24.7 seconds <br />
*** New: 10.5 seconds (Except first time that adds 8 sec)<br />
**** The parsing that calls is_abstract (that tries to instanciate all vtk classes) is now only run the first time.<br />
**** get_items_from_sigstring takes 2 seconds, maybe we can use a lookup dict for already computed sigstrings?<br />
** Now using a general python function wrapper<br />
*** VTK classes are wrapped into python function that does not depend on vistrails<br />
*** VTK functions can be executed without vistrails<br />
*** The spec maps functions into vistrails modules, but can also describe wrapping <br />
*** A general python function wrapper that supports <br />
**** kwarg inputs<br />
**** single, list, dict outputs<br />
**** callback for progress reporting<br />
**** temporary file generator for using FilePool<br />
**** optional output generation<br />
*** Creating specs:<br />
***** Create spec by hand<br />
***** Auto-create spec outline (TODO) and manually finish it<br />
***** Dynamically create spec (VTK)<br />
***** Implement documentation wrappers (Can use scikitlearn wrapper to wrap numpydoc) (TODO)<br />
***** Classes as bad functions needs to be wrapped in new functions before they are wrapped. This is different for each package.<br />
****** Classes is hard: Like VTK, and matplotlib. Scikit-learn does still not wrap classes<br />
***** Spec diffing and patching could be done using code from matplotlib.<br />
** Still needs upgrades from old VTK package<br />
*** Is it possible to dynamically wrap functions, e.g, you see a SetFunc and just remove the 'Set' prefix. Or do you need to create a complete mapping?<br />
<br />
=== February 11, 2015 ===<br />
'''Updates'''<br />
* Update from Friday's meeting<br />
** discussed VisTrails internals<br />
** discussed wrapping<br />
*** xml discussion, hard to modify because tied to db code<br />
*** TE has made it possible to add the schema-defined attributes to the intermediate representation<br />
*** higher-level operations on the port specs<br />
** make sure the simple case works<br />
*** [JF] take a simple package with documentation and figure out what the base case for wrapping is<br />
<br />
'''Items to Discuss'''<br />
* [TE] VTK wrapping<br />
** Dynamic loading works<br />
** Reading XML is fast enough, but [https://gist.github.com/rexissimus/8272d3c776ccfa115d82 unserializing data is slow]<br />
** Working on patterns for patching<br />
** Matplotlib has many advanced patterns like argument ordering, nested arguments, alternateSpecs, output types.<br />
*** Having all this in a general wrapper might confuse users?<br />
** [RR] Delay module (except for identifiers) until you need it---e.g. don't deal with port specs, etc. until necessary<br />
* Scripting Support [https://github.com/VisTrails/VisTrails/pull/950 #950]<br />
** [RR] Issue with getting code from modules<br />
** Design a simple solution<br />
** [JF] Couldn't you use modules as black boxes without conversion, just to call into modules/subworkflows easily from e.g. IPython?<br />
*** [RR] This is a job for the API, and a very separate use case. See [https://github.com/VisTrails/VisTrails/pull/24 #24]<br />
<br />
=== February 4, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* Wrapping<br />
** Format to use? Currently XML (like current matplotlib)<br />
*** JSON and YAML have simple "to python dictionary" methods<br />
*** But don't stream<br />
*** YAML a lot easier for humans<br />
** [DK] vtk-new-package also changes parameter names, creates enumerations<br />
*** intermediate schema needs to be extensible<br />
*** packages will want to store there specific infos for compute() method generation<br />
*** also might have specs-altering info, like matplotlib's alternateSpec<br />
** representation to code , registry already has schema for some aspects<br />
** [RR] We might want to see if Module subclasses can be created lazily<br />
*** no need to create all the classes just to register them in the registry and never actually use most of them<br />
*** future effort<br />
* [RR] Where should VisTrails packages live?<br />
** tej installs as 'vistrailspkg.tej', TE installed it as 'userpackages.tej'<br />
** Currently, standard packages are 'vistrails.packages.', user packages are 'userpackages.' and packages loaded through pkg_resources might be anything<br />
** [RR] Use 'vistrailspkg.' everywhere?<br />
** Long-term effort to simplify package distribution/installation (and have VisTrails get them automatically?)<br />
<br />
=== January 28, 2015 ===<br />
'''Updates'''<br />
* T. Caswell to come visit on Fri 6 to discuss wrapping work<br />
<br />
'''Items to Discuss'''<br />
* [TE] New VTK wrapping<br />
** Current code by DK seems a good deal faster<br />
** Generates XML that can be patched/tweaked, generates Python code from it, VisTrails only loads generated Python code<br />
** RR would rather have VisTrails load intermediate representation (= XML) directly, wants to make sure this is not slower<br />
** The goal is to turn the intermediate step into something generic that would be used for every wrapped package (vtk, numpy, matplotlib, sklearn, java) instead of each having its own<br />
** TC has its own code at [https://github.com/Nikea/VTTools/blob/master/vttools/wrap_lib.py github:VTTools] which parses numpy docstrings and generates modules, doesn't yet handle classes or persist anything<br />
* Web crawler<br />
** Right now, TE starts jobs for "start crawler", "stop crawler", "install classifier"<br />
** RR would rather have the crawling be a job as far as VisTrails and tej are concerned<br />
** The whole thing would be one pipeline: load examples, train classifier, start crawler [check for job, kill previous one, upload model, start processes], get snapshot, visualize<br />
*** Need some support in tej and job submission system: long-running jobs, stop a job (wait for it to finish?), restart a job even though results are cached<br />
<br />
=== January 21, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Unified wrapping method discussion [https://github.com/VisTrails/VisTrails/issues/991 #991]<br />
** TE to work on reusable method with intermediate representation, starting with VTK<br />
* [RR] Examples for scikit-learn: JF has an old example using Weka with parameter exploration (not currently in source tree)<br />
** AM's examples are enough<br />
* [AM] scikit-learn package is done, merge it in? [https://github.com/VisTrails/VisTrails/pull/955#issuecomment-70286788 #955]<br />
** RR will merge<br />
* [RR] What should copyright headers say? [https://github.com/VisTrails/VisTrails/pull/994#issuecomment-70173935 #994]<br />
** Let's keep everything in there: Utah/Poly/NYU<br />
<br />
=== January 14, 2015 ===<br />
'''Updates'''<br />
* [TE] Working on classifier<br />
* [RR] Scripting integration, work in progress<br />
<br />
'''Items to Discuss'''<br />
* [RR] Unified wrapping method discussion ([https://github.com/VisTrails/VisTrails/issues/991 #991])<br />
** Let's talk next week, [AM] and [DK] are not here<br />
<br />
=== January 7, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* make sure that we address critical issues, questions, and pending review branches in a timely manner<br />
* scripting support<br />
** [RR] no issues if we want to just keep annotations in the generated code to allow the link back to a workflow<br />
** [RR] can translate from workflow to script, working on script to workflow<br />
** will work for parameter value changes, structural changes require changes to the annotations<br />
** need to publish best practices here<br />
** would be cool to do looping in scripts (easier interface than with workflows)<br />
* notebook support (convert form notebook to workflow)<br />
** RR will sync with FC on this<br />
* Issue with console in built-from-scratch<br />
** [TC] iPython rearranged some of the completion stuff in 2.2 and 2.3<br />
** binary has old version of iPython -> 1.0.0, should we update?<br />
* [TC] automated wrapping of numpy and scipy<br />
** discovered a bunch of malformed documentation in numpy and scipy<br />
** has github repo for vistrails tools<br />
** example modules wrap a bunch of R stuff (not baked in, just how things are)<br />
** will be pushing wrapping logic up<br />
** port names forbidden (window and domain)<br />
** have an import hook to get from yaml directly to VisTrails Modules<br />
** should work for any python modules with well-formed numpy docstrings.<br />
* [Action] should make it clear in documentation that Constant now means serializable not that the value doesn't change (e.g. List)<br />
* [TC] might be interesting to try to build components of matplotlib and accumulate in figure (long-term project, but thinking about how this might work)<br />
* [TE] build and build scripts<br />
** completely automatic, buildbot<br />
** need to set the build machines for the environment we want for the binary<br />
** would virtualenv work here?<br />
** [TC] anaconda can pin versions, potential path to test different configurations<br />
** Q: upload nightly binary builds? A: makes sense, make sure they are well-labeled<br />
* sourceforge stats: e.g. http://sourceforge.net/projects/vistrails/files/vistrails/nightly/vistrails-src-nightly.tar.gz/stats/timeline?dates=2014-01-01+to+2015-01-07<br />
* package issues (see Remi's message)<br />
* [TE] Scope of tej<br />
** Support single ssh commands?<br />
** Queue can be used as a remote machine (crawler is using queue.call*)<br />
* SourceForge stats: http://sourceforge.net/projects/vistrails/files/vistrails/nightly/vistrails-src-nightly.tar.gz/stats/timeline?dates=2014-01-01+to+2015-01-07<br />
<br />
== Older meetings ==<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=11862Development2016-05-25T14:41:54Z<p>Remi: /* May 25, 2016 */</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2016 ==<br />
<br />
=== May 25, 2016 ===<br />
<br />
'''Updates'''<br />
* Mailing-list fixed<br />
* Getting stats from the "server news" mechanism (via [https://gist.github.com/remram44/7b955a0a2cfa9382f89702531d1fe6f8 parser])<br />
<br />
'''Items to Discuss'''<br />
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)<br />
<br />
=== May 18, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] mailing list still down - Julio working on it<br />
* [TE] Improved union port<br />
** New schema 1.0.5 adds portitem.union string attribute<br />
** Works with pipeline view, parameter exploration view, and mashup view<br />
** Should work well for matplotlib<br />
** Use with VTK ports?<br />
*** It uses [item_1,item_2] style ports usually with different tuple length.<br />
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]<br />
<br />
=== May 11, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?<br />
*** Shows single port for all ports with same sort_key<br />
*** Select specific type when adding/showing<br />
*** Only one type should be visible on the module at any time<br />
*** Making tuple parameters union types is probably too complicated<br />
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly) <br />
<br />
* [TE] Feature suggestion: Let connections connect to functions/parameters<br />
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters<br />
** Use separate views for ports and functions (like before)<br />
*** Allow rearranging of functions<br />
** Pros:<br />
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.<br />
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.<br />
**** Makes List module simpler<br />
*** Connect to parameter in tuple<br />
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds. <br />
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.<br />
** Cons:<br />
*** Interface more complicated and crowded<br />
*** More difficult for users to grasp?<br />
*** No immediate use case<br />
*** Makes no union type for tuple parameters a bigger problem.<br />
<br />
* [TE] Publish user survey?<br />
** No response from vistrails release yet<br />
** Not announced on vistrails-users yet.<br />
<br />
=== May 4, 2016 ===<br />
'''Updates'''<br />
* VisTrails 2.2.4 released<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
<br />
=== April 27, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Wrapping Bokeh<br />
** New method type "operation"<br />
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output<br />
**** Glyph sets function name and arguments as output<br />
***** Not serializable, but should always be computed together with the subject of the operation.<br />
**** Figure applies Glyph operation on itself<br />
*** Useful for matplotlib?<br />
**** matplotlib currently find actors by searching upstream?<br />
**** But this is more difficult since matplotlib is not function-based.<br />
** New "union" port type<br />
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value<br />
**** Uses only first value in tuple<br />
**** Bokeh have many properties that support (list, string, Float)<br />
*** Does not work with depth=1 types (e.g. Float of depth 1)<br />
**** Turn Depth 1 types with known type into lists?<br />
<br />
* [TE] New bugfix release<br />
** Finalize survey<br />
<br />
=== April 20, 2016 ===<br />
'''Updates'''<br />
* Merged upgrade fixes<br />
** Now has infinite loop test<br />
** loop test now correctly loops 5 times<br />
*** remap_module used wrong (current) package version when creating ModuleDescriptor<br />
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])<br />
<br />
'''Items to Discuss'''<br />
* New bugfix release<br />
** Merge usagestats<br />
** Need to finalize survey<br />
<br />
=== April 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?<br />
<br />
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])<br />
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]<br />
** Cherry-pick test to preserve source?<br />
** Need maxPipelineFixAttempts option?<br />
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.<br />
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?<br />
*** Would need to be moved into handle_invalid_pipeline<br />
<br />
=== April 6, 2016 ===<br />
'''Updates'''<br />
* [TE] Query View improvements<br />
** Done except for displaying "Version Info" and #1164<br />
* [RR] Working on finishing new interpreter (finally!)<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Try latest upgrade first, then previous.<br />
** If all fail, select latest upgrade?<br />
<br />
=== March 30, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Query View improvements<br />
** Unified do_version_switch and get_upgrades as validate_version()<br />
** Problems with query view using its own controller<br />
*** Fixed module info not being displayed for query results<br />
*** Still no version info for query results<br />
*** Still an issue with version query view not being updated<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== 2015 ==<br />
<br />
=== December 30, 2015 ===<br />
'''Updates'''<br />
* Bokeh package finished<br />
** Can output to Browser/IPython/Spreadsheet (Limited functionality in spreadsheet)<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Bokeh needs QWebEngine (Chromium) in Qt5 which requires PyQt5<br />
*** Many plotting libraries only work in browsers<br />
*** But Chromium may be a big and controversial dependency<br />
*** Or create a browser version of the spreadsheet?<br />
** Some users have requested Python 3<br />
** Not backwards compatible?<br />
** Target for VisTrails 3.0 with new interpreter?<br />
<br />
=== December 16, 2015 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Users list question<br />
* [TE] Wrapping Bokeh<br />
** Added bokeh property parser<br />
*** complements numpydoc parser<br />
*** Not all attributes are available, partially because I am using a pre-release version.<br />
** Wrapped modules<br />
*** bokeh.plotting finished (Backwards like matplotlib: Create plot then draw on it)<br />
*** bokeh.charts mostly done (Missing arguments)<br />
** QWebView mostly works (but no zooming or buttons)<br />
*** It would be better to use external browser or QWebEngineView<br />
** Added improvements to wrapper<br />
*** Generalized port translations (Color/Path to native types)<br />
*** Can use multiple docstring/property parsers simultaneously<br />
<br />
=== December 9, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Looking into wrapper<br />
** Will port TensorFlow<br />
** Doing a pass on the wrapper code & doc<br />
*** UX improvements<br />
** Factorize high-level docstring parsers into core.wrapper? (sphinx, numpy, google)<br />
** Problem with name vs module_name (fails sklearn tests because class_by_name() is broken)<br />
** Looking into compressing specs, lazy registration of modules<br />
<br />
=== December 2, 2015 ===<br />
'''Updates'''<br />
* [TE] Package wrapping<br />
** Wrapped most of numpy/scipy using __all__<br />
** Added class attribute and method access for classes (Can add to class or to separate inspector module)<br />
** TODO: Document wrapping procedure<br />
<br />
'''Items to Discuss'''<br />
<br />
=== November 25, 2015 ===<br />
'''Updates'''<br />
* [RR] New interpreter is coming along<br />
** Everything is a stream<br />
** Task system, eventually work-stealing parallelism<br />
** Streams no longer advance in lock-step<br />
*** Allows for constructions like filter, join, sorted-merge<br />
** depth>1 still to be tested, no current plan to allow this through the module interface<br />
*** Through looping though, can do streams of streams<br />
<br />
'''Items to Discuss'''<br />
* [TE] Package wrapping<br />
** Wrapped most numpy functions (no polynomials)<br />
** and half of scipy classes and functions<br />
** Added functions manually from documentation (no parseable lists?)<br />
** Focusing on data flow constructs (no property/method access for classes)<br />
*** Using spec diff (empty for now) and dynamic parser (stores spec in .vistrails/)<br />
** Python class/function wrapper now stable<br />
** Add google doc parser?<br />
*** Is there a parser available?<br />
<br />
=== November 18, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Package wrapping<br />
** Numpy class wrapping works<br />
*** Basic type is List, since most types are array_likes.<br />
*** TODO: More classes and functions<br />
*** Some operations are in-place and some aren't, and docstring not super clear<br />
*** [RR] It seems only methods mutate, and most have a numpy.xx function equivalent; just manually go through the methods and remove the mutating ones?<br />
** Class wrapper is modular<br />
*** Docstring parser and type string parser can be customized<br />
*** Classes can have optional attribute/method ports<br />
**** Inspectors and attribute/method modules can be created separately<br />
** Use PythonCalc as an example<br />
*** Can be re-implemented as a function with a parseable docstring<br />
*** [RR] PythonCalc is just an example, and it might actually make sense to get rid of it, or do a proper math package with scalar operations (as separate modules, no combobox)<br />
** [RR] TensorFlow to use wrapping as well<br />
*** Very simple (only types 'tensor' & 'variable')<br />
<br />
=== November 11, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Package wrapping<br />
** Wrapping numpy's classes using numpydoc (ndarray)<br />
*** Wrapping constructor arguments, attribute getters/setters as module ports<br />
**** Then how to access attributes afterwards?<br />
***** Use input value port to class modules?<br />
*** Wrap methods as modules? E.g., `ndarray.shape`.<br />
*** Function wrapping is almost a subset of class wrapping<br />
**** May be able to use the same parsing/execution methods<br />
** VTK's non-getter/setter methods could be wrapped as modules<br />
*** We could then remove extra logic in interpreter for keeping function order<br />
** Bokeh uses autogenerated docstrings<br />
*** We may be able to read the specification directly<br />
<br />
* [RR] [https://github.com/VisTrails/VisTrails/pull/1138 TensorFlow package]<br />
** Basic setup working, can execute the Mandelbrot example<br />
** Will autogenerate the operations<br />
<br />
=== November 4, 2015 ===<br />
'''Updates'''<br />
* [RR] Interpreter work requires a fix for [https://github.com/VisTrails/VisTrails/issues/1137 DB issue #1137]<br />
* [DK] Kitware's Resonant [http://www.kitware.com/source/home/post/173]<br />
** [https://github.com/girder/girder Girder]: data management system<br />
** [https://github.com/Kitware/romanesco Romanesco]: execution engine, uses [http://www.celeryproject.org/ Celery] for task management<br />
** Resonant Flow: web app for editing and executing workflows<br />
<br />
'''Items to Discuss'''<br />
* [TE] Added OSX Lion (10.7) VM on build machine using vagrant (seems ok with license?)<br />
** New builds works on Lion<br />
** Change the minimum requirement to 10.7?<br />
* [TE] Package wrapping<br />
* Added upgrade suggestions using 2 spec versions ([http://pastebin.com/30XUU8Gp example])<br />
** using name edit distance to find matches<br />
** Could use something better like port similarity for modules, and type similarity for ports<br />
** Should write python upgrade code?<br />
** Added spec to sklearn package<br />
*** Can now diff spec versions and keep the spec static<br />
*** There will be problems with downgrades and package versions<br />
<br />
=== October 29, 2015 ===<br />
'''Updates'''<br />
* [TE] nightly-build-mac fixed (Needed to approve xcode license)<br />
<br />
'''Items to Discuss'''<br />
* [TE] matplotlib wrapper<br />
** ported to general spec<br />
** Diff can be used on general spec<br />
*** Only needed minor changes<br />
*** Fixed indexing bug that corrupted specs<br />
*** Can now move diff tools to core/wrapper<br />
** Add new plots?<br />
*** New plots in mpl 1.3: eventplot<br />
*** New plots in mpl 1.4: angle_spectrum, magnitude_spectrum, phase_spectrum', violinplot<br />
*** No new plots in mpl 1.5 (from looking at boilerplate.py)<br />
<br />
=== October 21, 2015 ===<br />
'''Updates'''<br />
* [RR] Alexis has arrived, will be working with [RR] on a more efficient interpreter<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1133 New matplotlib package]<br />
** Supporting multiple package versions<br />
*** Check which version can be loaded [https://github.com/VisTrails/VisTrails/pull/1135 #1135]<br />
*** We cannot show version requirements in the list of packages because old packages are loaded when importing the codepath<br />
** Visual diff does not work well, but we can use the spec differ to see differences.<br />
** Unifying vtk and matplotlib wrapping specs (WIP)<br />
*** And porting matplotlibs spec differ to work on the general spec<br />
*** Spec differ can then be used on vtk and other packages in the future <br />
*** Will enable automatic upgrade generation<br />
* [RR] Rework interpreter<br />
** restore abstraction between interpreter/module code<br />
** build looping, streaming, caching into the interpreter<br />
** rework cache<br />
** look into parallelism once basic functionality is there<br />
<br />
=== October 14, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1133 New matplotlib package] (adding numpydoc parser)<br />
<br />
=== October 7, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://github.com/VisTrails/VisTrails/tree/fix-matplotlib-parser New matplotlib parser] (adding numpydoc parser)<br />
** Added numpydoc attribute parser for plots.<br />
*** TODO: Need better port spec reconciliation with call signature parser.<br />
*** We can create a general numpydoc parser, but attribute types are unique to matplotlib.<br />
** Needs package versioning<br />
*** New spec will not support matplotlib < v1.4 due to changed path to axes classes.<br />
*** Load spec version corresponding to installed matplotlib version?<br />
*** Need version downgrades?<br />
<br />
=== September 30, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Updating Matplotlib parser<br />
** Matplotlib docstring parser fails on numpy docstrings<br />
** I have [https://github.com/VisTrails/VisTrails/commit/eae8b0c16260810587e47c41cab53552cbb3cf0a added basic numpydoc parsing]<br />
*** Only used by a few docstrings so far<br />
*** Parsing uses many sources (class tables, signatures, ACCEPTS, method docstrings, definition parsing).<br />
** Caswell said they were thinking about moving to traitlets, but this is not ready yet<br />
<br />
=== September 23, 2015 ===<br />
'''Updates'''<br />
* [TE] VisTrails 2.2.3 released<br />
** Also have pushed to PyPI, binstar, etc.<br />
** Sourceforge vs. GitHub<br />
*** should be able to host releases on GitHub<br />
*** nightly binaries pushed to sf each night<br />
*** old binaries?<br />
* [TE] Re-wrapping MatPlotlib<br />
** Keep static generation - Docstrings are brittle beween matplotlib versions<br />
** Keep generating the executable classes - but create functions, not vistrails Modules <br />
*** Why is there so much patching?<br />
** Re-implemented Alternate PortSpec for InputPortSpec<br />
*** It will now inherit specs from parent (No need to reimplement)<br />
** Module upgrades?<br />
<br />
'''Items to Discuss'''<br />
<br />
=== September 16, 2015 ===<br />
'''Updates'''<br />
* [TE] Job Monitor tests and documentation done<br />
** OK to leave document package in packages directory?<br />
<br />
'''Items to Discuss'''<br />
* [TE] Library wrapping: How to do code patching? [https://docs.google.com/drawings/d/1Sh3eOXot6v2y880jHKQrqXqAwr-0aoqYligg3z8G8MY/edit?usp=sharing Wrapping diagram]<br />
** Insert code into module template (Old VTK code)<br />
** Patch the library that is called (New VTK code)<br />
** Somehow store code in spec and apply when executed?<br />
*** Executed code needs at a minimum access to inputs, outputs, current class<br />
*** Use standard input/output dict and obj reference that the code operates on?<br />
*** Can this be done while keeping the execution abstract?<br />
<br />
* Do new release now?<br />
<br />
=== September 9, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* New release?<br />
** [RR] wants reprounzip<br />
** tej changes & doc<br />
** job monitor & jobmixin fixes<br />
** mongodb<br />
** reprounzip<br />
** warning: MplFigure type is int<br />
** tabledata: add Reshape, DictoToTable & ListToTable, fix ListToTable with numpy arrays<br />
** PythonSource can have same name for an input & output<br />
** don't show spreadsheet at exit<br />
<br />
=== September 2, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Limit autosaves? [https://github.com/VisTrails/VisTrails/pull/1126 #1126]<br />
<br />
=== August 26, 2015 ===<br />
'''Updates'''<br />
* [TE] Added Job support to Parameter Explorations (Requested by Colin), and Mashups<br />
** specify job ids, need to specify different ids for parameter explorations since they have the same version id<br />
** how to deal with parameters passed in on command line<br />
* [General] provenance: should be creating a new version when we execute workflow with changed parameters?<br />
** currently, custom_params annotation stores this in provenance currently<br />
** [RR] API doesn't record provenance if passing in parameters: http://git.io/vsAA5<br />
'''Items to Discuss'''<br />
<br />
=== August 19, 2015 ===<br />
'''Updates'''<br />
* [RR] Improvements to tej, users' guide entry & example [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
* [RR] Internal docs [https://github.com/VisTrails/VisTrails/pull/1116 #1116]<br />
** On ReadTheDocs: [http://vistrails.readthedocs.org/ vistrails.readthedocs.org]<br />
* [TE] Working on Job Monitor documentation<br />
<br />
'''Items to Discuss'''<br />
* [RR] JobMixin and JobMonitor: stable now?<br />
** Definitely needs more tests<br />
* [TE] Stop testing VisTrails 2.0?<br />
<br />
=== August 12, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Internal docs [https://github.com/VisTrails/VisTrails/pull/1116 #1116]<br />
** On ReadTheDocs: [http://vistrails.readthedocs.org/ vistrails.readthedocs.org]<br />
* [RR] JobMixin and JobMonitor: stable now?<br />
** Definitely needs more tests<br />
* [RR] Improvements to tej, users' guide entry & example [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
* [RR] Build broken on Travis; because of IPython 4 released today? ([https://travis-ci.org/VisTrails/VisTrails/builds/75279795 build 992]; [https://github.com/VisTrails/VisTrails/issues/1123 #1123])<br />
<br />
=== August 5, 2015 ===<br />
'''Updates'''<br />
* [TE] Fixes to Jobs<br />
** Could not delete jobs<br />
** Could not run job in group<br />
** Job not reset when calling ModuleSuspended<br />
** Added deleting job from context menu<br />
* [RR] Writing documentation for everything [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
<br />
'''Items to Discuss'''<br />
<br />
=== July 29, 2015 ===<br />
'''Updates'''<br />
* RR still looking into new interpreter thing<br />
** Goal is to take out scheduling logic from Module so it can be split in multiple processes, and so that smarter strategies can be added in time<br />
** This means some work on packages<br />
** Spreadsheet can live in kernel process? Still some UI stuff that will take work (changing configuration, persistent archive's viewer, ...)<br />
* [TE] Problems running examples [https://github.com/VisTrails/VisTrails/pull/1111 #1111]<br />
** Testing of more examples requires additional packages on the test machines.<br />
** Fixed faulty line-ending in PythonSource:s failing on Python 2.6.<br />
** Test suite now testing SUDSWebServices (If web service is down, test suite will fail)<br />
** preferences.py test failed reloading 'dialogs' package, switched to using 'URL'.<br />
<br />
'''Items to Discuss'''<br />
<br />
=== July 22, 2015 ===<br />
'''Updates'''<br />
* [TE] mailing lists back online<br />
* [TE] Working on [https://github.com/VisTrails/VisTrails/issues/1107 #1107]<br />
* [RR] Working on ReproZip package <br />
<br />
'''Items to Discuss'''<br />
* [RR] MongoDB package [https://github.com/VisTrails/VisTrails/pull/1106 #1106]<br />
* [RR] Example for tej docs? [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
<br />
=== July 15, 2015 ===<br />
'''Updates'''<br />
* 2.2.2 released<br />
<br />
'''Items to Discuss'''<br />
* RR is considering executing everything in an IPython kernel (i.e. separate Python interpreter, like the one spawned for a notebook)<br />
** It's a separate process that we can restart without restarting the app/gui<br />
** We can isolate the execution environment (e.g. for the server)<br />
** We can run the whole thing remotely (if your data is elsewhere, just run VisTrails locally and the pipeline elsewhere)<br />
** Pipeline execution no longer makes the interface hang, it just makes the kernel hang (but that's fine)<br />
** We can use notebooks as modules (probably way nicer than the PythonSource module)<br />
** We can run multiple kernels so long as the ports carry things that are serializable<br />
*** meaning we can put the multithreaded-interpreter without all the hacky parts it has now<br />
*** we can run IPython kernels in all the languages IPython supports, [https://github.com/ipython/ipython/wiki/IPython-kernels-for-other-languages currently 46]<br />
<br />
=== July 8, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Buildbot github hook not working after IP address change<br />
* [TE] No reference to VistrailsApplication ([https://github.com/VisTrails/VisTrails/issues/1103 #1103])<br />
** [RR] Added PR [https://github.com/VisTrails/VisTrails/pull/1104 #1104]<br />
* [TE] Reopening VT file after saving with bundled subworkflow won't offer subworkflow upgrade ([https://github.com/VisTrails/VisTrails/issues/1102 #1102])<br />
** allow manual delete to fix right now<br />
** fix this on top of the use-uuid branch<br />
* [TE] Release VisTrails 2.2.1? ([https://github.com/VisTrails/VisTrails/blob/v2.2/CHANGELOG CHANGELOG])<br />
** [https://github.com/VisTrails/VisTrails/pull/1093 output-modules-use-global-dir]<br />
** [https://github.com/VisTrails/VisTrails/milestones/version%202.2 Current issues for 2.2]<br />
<br />
=== July 1, 2015 ===<br />
'''Updates'''<br />
* [TE] PROV fixed<br />
* [TE] Working on subworkflow issues<br />
* [RR] Considering reworking the controller (log vs exception problem, retained upgrades causing interferences) and also the interpreter (IPython?)<br />
<br />
'''Items to Discuss'''<br />
<br />
=== June 24, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* UV-CDAT<br />
* [TE] Can a cyclic workflow be valid? ([https://github.com/VisTrails/VisTrails/issues/1097 #1097])<br />
** focus on disabling the ability to create cyclic pipelines because more things break than just this with a cyclic pipeline<br />
* [RR] Relative paths ([https://github.com/VisTrails/VisTrails/pull/1057 #1057])<br />
** This interacts with the new bundle; how do handle packing files inside the VT bundle?<br />
<br />
=== June 17, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] current_version and reusing existing upgrades are broken ([https://github.com/VisTrails/VisTrails/issues/1095 ticket #1095])<br />
** Could be that export to PROV is using an unflushed upgrade pipeline?<br />
** current_version would then be correct.<br />
** It may work to flush the actions before exporting?<br />
<br />
=== June 10, 2015 ===<br />
'''Updates'''<br />
* DAT: fixed VTK issue on Linux and Mac<br />
* Still crashes on Windows. Need help! Reminder: this works in the VisTrails spreadsheet (QCellPresenter), although no widget get changed there during a drag<br />
** Is it a VTK bug?<br />
** Is it simply impossible to change widgets during the drag, should we do it a different way?<br />
** Did I miss something that is done in VisTrails but somehow not in DAT?<br />
** Low prio, UV-CDAT doesn't run on Windows anyway<br />
<br />
'''Items to Discuss'''<br />
* [TE] Executable Paper ([https://github.com/VisTrails/VisTrails/issues/1086 ticket #1088] [https://github.com/VisTrails/VisTrails/pull/1088 Updated pull request])<br />
* Single instance code and batch mode<br />
<br />
=== June 3, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Executable Paper ([https://github.com/VisTrails/VisTrails/issues/1086 ticket #1088] [https://github.com/VisTrails/VisTrails/pull/1088 pr])<br />
** Requires fixes to command line parameters, Output modules, and batch mode<br />
** How to test this<br />
** Updated missing/outdated flags<br />
** Fixed view issues when generating graphs<br />
* [TE] batch mode<br />
** SpreadsheetOutput not enabled in batch mode. Should we check is_running_gui instead?<br />
** Other instance setting flags from caller<br />
** Is graphsAsPdf replacing spreadsheetDumpPdf?<br />
** graphsAsPdf true by Default?<br />
** Batch mode executing by default (Not needed when generating graphs)<br />
** Re-added workflowInfo as withWorkflowInfo for writing graph and xml workflow<br />
** [DK] batch mode should be outputting to files or stdout, shouldn't always trigger SpreadsheetMode<br />
** execute flag, maybe make execution the default and allow a "--no-execute" if you only want to capture graphs, for example<br />
<br />
=== May 27, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] API question<br />
** http://lists.vistrails.org/pipermail/vistrails-users/2015-May/000357.html<br />
* Status of DAT<br />
** Can add VCS plot to DAT but need to work on configuration windows<br />
** Documentation for graphics templates for UV-CDAT/vcs?<br />
* Qt support?<br />
<br />
=== May 20, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Reviving the DAT, integrating scripting & porting UV-CDAT!<br />
** On [https://github.com/VisTrails/DAT GitHub] ([https://github.com/VisTrails/DAT/issues issues])<br />
** Merging 2 years of development taking longer than expected, but getting there. The plan is to get the patches in VisTrails and never fork again, we never want to get in UV-CDAT's situation (and don't need to).<br />
** VTK cell works fine on Linux but there was flickering on Mac & Windows before; still issues on Mac (Windows status unknown)<br />
** Can get a VCS plot soon (but will need VTK cell fix)<br />
** Integrate in UV-CDAT's build system (so we have cdms, VCS, ...) -> RR can do this, low priority<br />
** How do we integrate scripting?<br />
*** We want to be able to seamlessly make changes to a plot by changing Python code<br />
*** Define new plots by entering Python code without writing modules/packages?<br />
* [RR] UV-CDAT [https://github.com/UV-CDAT/uvcdat/issues/1275 needs] [https://github.com/VisTrails/VisTrails/pull/1073 #1073], please take a look<br />
* [RR] Subworkflow issues<br />
** [https://github.com/VisTrails/VisTrails/issues/1065 #1065]: missing attribute (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/1066 #1066]: groups in subworkflows were ignored when finding dependencies (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/725 #725]: "upgrade subworkflow" button not popping up (fixed); also, namespace issue<br />
** [https://github.com/VisTrails/VisTrails/issues/1071 #1071]: subworkflows don't go through upgrades<br />
* [RR] [https://github.com/VisTrails/VisTrails/issues/1074 #1074]: can't load and edit a single pipeline if upgrade happen<br />
<br />
=== May 13, 2015 ===<br />
'''Updates'''<br />
* [RR] Test skipping whitelist ([https://github.com/VisTrails/VisTrails/pull/1069 #1069]) -- low priority<br />
* [RR] Custom matplotlib modules can't be compatible with both 2.1 and 2.2 ([https://github.com/VisTrails/VisTrails/issues/1067 #1067]); should be fixed for ALPS ([https://github.com/VisTrails/VisTrails/issues/1070 #1070])<br />
<br />
'''Items to Discuss'''<br />
* [RR] UV-CDAT [https://github.com/UV-CDAT/uvcdat/issues/1275 needs] [https://github.com/VisTrails/VisTrails/pull/1073 #1073], please take a look<br />
* [RR] Subworkflow issues<br />
** [https://github.com/VisTrails/VisTrails/issues/1065 #1065]: missing attribute (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/1066 #1066]: groups in subworkflows were ignored when finding dependencies (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/725 #725]: "upgrade subworkflow" button not popping up (fixed); also, namespace issue<br />
** [https://github.com/VisTrails/VisTrails/issues/1071 #1071]: subworkflows don't go through upgrades<br />
* [RR] [https://github.com/VisTrails/VisTrails/issues/1074 #1074]: can't load and edit a single pipeline if upgrade happen<br />
<br />
=== May 7, 2015 ===<br />
'''Updates'''<br />
* [RR] Export/import workflow to Python working!<br />
<br />
'''Items to Discuss'''<br />
* [Claudio] UV-CDAT<br />
** The UV-CDAT project is the biggest user base of VisTrails<br />
** VisTrails package management provides a lot of friction towards people plugging in their code<br />
** Need to make it easy to integrate your random Python scripts in the system without having to deal with all the boilerplate, at least in the first step<br />
** [RR] argues that modules are still good; UV-CDAT shouldn't move towards a purely script-based backend<br />
** [RR] export/import with Python could reduce a lot of that friction by allowing 1) to edit workflow as Python 2) to open up boxes automatically if needed code doesn't match actual modules<br />
** ...<br />
<br />
=== April 29, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] <strike>Abstractions</strike> subworkflows status ([https://github.com/VisTrails/VisTrails/labels/subworkflows tickets])<br />
* [RR] matplotlib compatibility (2.1 & 2.2), [https://github.com/VisTrails/VisTrails/issues/1067 #1067]<br />
** RR to try and fix ALPS matplotlib modules<br />
<br />
=== April 22, 2015 ===<br />
'''Updates'''<br />
* 2.2.0 has been released!<br />
* Windows issue (via email from Ryan)<br />
** issue with manifest file (may be a new file in VTK6?)<br />
** Tommy has regenerated new Windows builds<br />
* Binaries, pypi, and conda released<br />
* [RR] Export as script<br />
* Python sources using VTK need to switch to SetInputData (users should be aware of this)<br />
<br />
=== April 15, 2015 ===<br />
'''Updates'''<br />
* Ready for 2.2.0 (apart from binary/deps issues)<br />
** Missing some libs (scikit-learn, tej, tdparser, SQLAlchemy+connectors<br />
** Windows: runvistrails.py is no longer used, so the PATH is wrong<br />
** Windows: pip is broken, but it probably wouldn't work anyway because of permission issues (disable this?)<br />
* Queries, upgrades and getPipeline() usage ([https://github.com/VisTrails/VisTrails/issues/1054 #1054])<br />
** Getting a pipeline with getPipeline() is not safe: it might return an invalid pipeline<br />
** This is used in many places throughout the code, like queries<br />
** Upgrading would require going through the controller, but that creates new actions<br />
* [TC] Avoid copying a module's output if it's used as input by exactly one downstream module ([https://github.com/VisTrails/VisTrails/issues/1060 #1060]) (useful for big numpy arrays you can update in-place)<br />
<br />
=== April 8, 2015 ===<br />
'''Updates'''<br />
* [DK] Merged RR's changes for output modules (1012 and 1013)<br />
** RR will merge remaining changes, then create v2.2 branch!<br />
<br />
'''Items to Discuss'''<br />
* BNL need numpy array to VTK image<br />
** Looks like VTK has some helpers for this<br />
** We will help if issues arise<br />
** Will contribute back to VisTrails package<br />
* Upgrade issue: [https://github.com/VisTrails/VisTrails/issues/1017 #1017]<br />
** Automatic upgrades should happen between versions of provided upgrades<br />
** Our existing upgrades work around this so it doesn't need to be 2.2.0<br />
* Corner-case VTK modules<br />
** No longer need VTKCell input port, so don't interfere with registry and API anymore<br />
** Still work weirdly, people probably shouldn't use them<br />
** But we have lots of clunky modules since we wrap the whole of VTK; some people might rely on this and know how to use them, let's keep them anyway<br />
** Ready for 2.2.0<br />
<br />
=== April 1, 2015 ===<br />
'''Updates'''<br />
* [RR] UV-CDAT: bugfixing for 2.2, long-term plans: implement scripting import/export in VisTrails, port to UV-CDAT<br />
* Possibly, try to move to regular VisTrails to use new features<br />
<br />
'''Items to Discuss'''<br />
* 2.2 release: nothing much is pending anymore, release next week?<br />
** [TE] vtkExporter classes [https://github.com/VisTrails/VisTrails/issues/1032 #1032]<br />
** [RR] Hiding upgrades in version tree might make it (or might be 2.2.1 so we can test it out) [https://github.com/VisTrails/VisTrails/issues/949 #949]<br />
** Output modules changes to go in<br />
* add note to documentation about order of parameters in VTK<br />
* add issue about exporter upgrades if not already there<br />
<br />
=== March 25, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* 2.2 release<br />
** Checklist on Github: https://github.com/VisTrails/VisTrails/wiki/2.2-Release-Checklist<br />
** Review output modules<br />
** RR has a couple more issues to fix<br />
** Ready to go -- sign app package for OSX? [https://github.com/VisTrails/VisTrails/issues/984 #984] We need access to the Apple Membership team<br />
<br />
=== March 18, 2015 ===<br />
'''Updates'''<br />
* [TE] VTK6 works<br />
<br />
'''Items to Discuss'''<br />
* [RR] Release v2.1.5 with backported tabledata?<br />
** MTA example needs updated tabledata (for JoinTables)<br />
** Google Maps package still not available<br />
* [RR] Work torward v2.2.0?<br />
** changes:<br />
*** new persistence<br />
*** API changes<br />
*** output module changes (upgrades?), maintain cells but try to upgrade<br />
*** not wrapping stuff<br />
*** VTK6? yes<br />
*** JobSubmission stuff?<br />
*** relabeling for upgrades [https://github.com/VisTrails/VisTrails/issues/949 #949]<br />
** makes sense, needs the tree view code to be updated, check selection<br />
** See [https://github.com/VisTrails/VisTrails/wiki/2.2-Release-Checklist 2.2 checklist]<br />
* Discussion of [https://github.com/VisTrails/VisTrails/issues/1016 #1016]<br />
** plumbing between outputs and output modes, how to define a mode that works for many outputs without writing for each output?<br />
<br />
=== March 11, 2015 ===<br />
'''Updates'''<br />
* [RR] persistent_archive done; merge? ([https://github.com/VisTrails/VisTrails/pull/755 #755])<br />
** note about the focus events for widgets<br />
** TE be aware of file_archive for future binaries that include persistent_archive<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/998 New VTK package] finished<br />
<br />
'''Items to Discuss'''<br />
* [RR] Unmark UV-CDAT/VisTrails as a fork of VisTrails/VisTrails? (see last week; decision needed)<br />
** RR email to JF about this<br />
** yes; email sent to Github<br />
* [RR] Switching order of output ports ([https://github.com/VisTrails/VisTrails/pull/1006 #1006])<br />
** added port specs are sorted at a separate spot (Module.*_port_specs properties) than those in the registry (which are sorted in the registry), but those two lists are just combined without respect to sort keys<br />
** need to determine whether the two lists should be merged or remain distinct<br />
** should make sure that order of input ports and output ports makes parallel connections for things with same order<br />
** DK suggests breaking backward compatibility here: workflows still run, can fix easily if a problem in an existing package.<br />
* [RR] Question about output modes ([https://github.com/VisTrails/VisTrails/pull/1007 #1007]), how to integrate in API ([https://github.com/VisTrails/VisTrails/pull/24 #24])<br />
** Should ImageFileMode be removed? ("image" is not a mode, "file" is)<br />
** ImageOutput missing?<br />
** Feel free to change how formats works<br />
* [TE] Test suite segfaults on Fedora 17 virtualbox. Install newer version? (Support ended in 2013)<br />
<br />
=== March 4, 2015 ===<br />
'''Updates'''<br />
* [TE] VTK wrapper<br />
** Works on VTK 5.10<br />
** Still need to test VTK 6<br />
** New general wrappers for python functions and classes <br />
<br />
'''Items to Discuss'''<br />
* [RR] Unmark UV-CDAT/VisTrails as a fork of VisTrails/VisTrails?<br />
** When [http://i.imgur.com/UTY2qfg.png filing a pull request] from UV-CDAT/VisTrails, VisTrails/VisTrails is selected by default<br />
** People keep forgetting to change the default ([https://github.com/VisTrails/VisTrails/pull/956 #956] [https://github.com/VisTrails/VisTrails/pull/968 #968] [https://github.com/VisTrails/VisTrails/pull/999 #999] [https://github.com/VisTrails/VisTrails/pull/1000 #1000] [https://github.com/VisTrails/VisTrails/pull/1003 #1003] [https://github.com/VisTrails/VisTrails/pull/1004 #1004] [https://github.com/VisTrails/VisTrails/pull/1005 #1005])<br />
** Only way to change that is to not have it marked as a fork of VisTrails/VisTrails<br />
** Github staff [https://gist.github.com/remram44/b364c24ef1e462ce6e2f can make that change for us]; should we do it?<br />
** Juliana: comments on this? visibility vs. convenience/annoyance for developers<br />
* [RR] What about [https://github.com/VisTrails/VisTrails/issues/991 #991]?<br />
<br />
=== February 25, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] vtkviewcell for infovis support, can we unify with VTKCell?<br />
** need to test this<br />
* [TE] vtk wrapping<br />
** Mostly finished<br />
** VTK 5.10 produce incorrect results with old wrapping<br />
*** Old wrapper is based mostly on VTK 4<br />
*** Most vtk_examples affected<br />
*** [http://www.vtk.org/Wiki/VTK/VTK_6_Migration/Replacement_of_SetInput]<br />
*** should be able to upgrade from SetInput to SetInputData (need to drop GetOutput and replace with self ports)<br />
*** can we change vtkInstance to just return self and not wrap things<br />
** terminator example not working under 5.8?<br />
* How does VTK wrapping fit into general wrapping framework?<br />
* [RR] new persistence package<br />
<br />
=== February 18, 2015 ===<br />
'''Updates'''<br />
* [RR] New VisTrails API and IPython integration ([https://github.com/VisTrails/VisTrails/pull/24 #24])<br />
<br />
'''Items to Discuss'''<br />
* [TE] VTK wrapping<br />
** [https://gist.github.com/rexissimus/16b74693545b5e7c1035 Benchmarking vtk package]<br />
*** Old: 24.7 seconds <br />
*** New: 10.5 seconds (Except first time that adds 8 sec)<br />
**** The parsing that calls is_abstract (that tries to instanciate all vtk classes) is now only run the first time.<br />
**** get_items_from_sigstring takes 2 seconds, maybe we can use a lookup dict for already computed sigstrings?<br />
** Now using a general python function wrapper<br />
*** VTK classes are wrapped into python function that does not depend on vistrails<br />
*** VTK functions can be executed without vistrails<br />
*** The spec maps functions into vistrails modules, but can also describe wrapping <br />
*** A general python function wrapper that supports <br />
**** kwarg inputs<br />
**** single, list, dict outputs<br />
**** callback for progress reporting<br />
**** temporary file generator for using FilePool<br />
**** optional output generation<br />
*** Creating specs:<br />
***** Create spec by hand<br />
***** Auto-create spec outline (TODO) and manually finish it<br />
***** Dynamically create spec (VTK)<br />
***** Implement documentation wrappers (Can use scikitlearn wrapper to wrap numpydoc) (TODO)<br />
***** Classes as bad functions needs to be wrapped in new functions before they are wrapped. This is different for each package.<br />
****** Classes is hard: Like VTK, and matplotlib. Scikit-learn does still not wrap classes<br />
***** Spec diffing and patching could be done using code from matplotlib.<br />
** Still needs upgrades from old VTK package<br />
*** Is it possible to dynamically wrap functions, e.g, you see a SetFunc and just remove the 'Set' prefix. Or do you need to create a complete mapping?<br />
<br />
=== February 11, 2015 ===<br />
'''Updates'''<br />
* Update from Friday's meeting<br />
** discussed VisTrails internals<br />
** discussed wrapping<br />
*** xml discussion, hard to modify because tied to db code<br />
*** TE has made it possible to add the schema-defined attributes to the intermediate representation<br />
*** higher-level operations on the port specs<br />
** make sure the simple case works<br />
*** [JF] take a simple package with documentation and figure out what the base case for wrapping is<br />
<br />
'''Items to Discuss'''<br />
* [TE] VTK wrapping<br />
** Dynamic loading works<br />
** Reading XML is fast enough, but [https://gist.github.com/rexissimus/8272d3c776ccfa115d82 unserializing data is slow]<br />
** Working on patterns for patching<br />
** Matplotlib has many advanced patterns like argument ordering, nested arguments, alternateSpecs, output types.<br />
*** Having all this in a general wrapper might confuse users?<br />
** [RR] Delay module (except for identifiers) until you need it---e.g. don't deal with port specs, etc. until necessary<br />
* Scripting Support [https://github.com/VisTrails/VisTrails/pull/950 #950]<br />
** [RR] Issue with getting code from modules<br />
** Design a simple solution<br />
** [JF] Couldn't you use modules as black boxes without conversion, just to call into modules/subworkflows easily from e.g. IPython?<br />
*** [RR] This is a job for the API, and a very separate use case. See [https://github.com/VisTrails/VisTrails/pull/24 #24]<br />
<br />
=== February 4, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* Wrapping<br />
** Format to use? Currently XML (like current matplotlib)<br />
*** JSON and YAML have simple "to python dictionary" methods<br />
*** But don't stream<br />
*** YAML a lot easier for humans<br />
** [DK] vtk-new-package also changes parameter names, creates enumerations<br />
*** intermediate schema needs to be extensible<br />
*** packages will want to store there specific infos for compute() method generation<br />
*** also might have specs-altering info, like matplotlib's alternateSpec<br />
** representation to code , registry already has schema for some aspects<br />
** [RR] We might want to see if Module subclasses can be created lazily<br />
*** no need to create all the classes just to register them in the registry and never actually use most of them<br />
*** future effort<br />
* [RR] Where should VisTrails packages live?<br />
** tej installs as 'vistrailspkg.tej', TE installed it as 'userpackages.tej'<br />
** Currently, standard packages are 'vistrails.packages.', user packages are 'userpackages.' and packages loaded through pkg_resources might be anything<br />
** [RR] Use 'vistrailspkg.' everywhere?<br />
** Long-term effort to simplify package distribution/installation (and have VisTrails get them automatically?)<br />
<br />
=== January 28, 2015 ===<br />
'''Updates'''<br />
* T. Caswell to come visit on Fri 6 to discuss wrapping work<br />
<br />
'''Items to Discuss'''<br />
* [TE] New VTK wrapping<br />
** Current code by DK seems a good deal faster<br />
** Generates XML that can be patched/tweaked, generates Python code from it, VisTrails only loads generated Python code<br />
** RR would rather have VisTrails load intermediate representation (= XML) directly, wants to make sure this is not slower<br />
** The goal is to turn the intermediate step into something generic that would be used for every wrapped package (vtk, numpy, matplotlib, sklearn, java) instead of each having its own<br />
** TC has its own code at [https://github.com/Nikea/VTTools/blob/master/vttools/wrap_lib.py github:VTTools] which parses numpy docstrings and generates modules, doesn't yet handle classes or persist anything<br />
* Web crawler<br />
** Right now, TE starts jobs for "start crawler", "stop crawler", "install classifier"<br />
** RR would rather have the crawling be a job as far as VisTrails and tej are concerned<br />
** The whole thing would be one pipeline: load examples, train classifier, start crawler [check for job, kill previous one, upload model, start processes], get snapshot, visualize<br />
*** Need some support in tej and job submission system: long-running jobs, stop a job (wait for it to finish?), restart a job even though results are cached<br />
<br />
=== January 21, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Unified wrapping method discussion [https://github.com/VisTrails/VisTrails/issues/991 #991]<br />
** TE to work on reusable method with intermediate representation, starting with VTK<br />
* [RR] Examples for scikit-learn: JF has an old example using Weka with parameter exploration (not currently in source tree)<br />
** AM's examples are enough<br />
* [AM] scikit-learn package is done, merge it in? [https://github.com/VisTrails/VisTrails/pull/955#issuecomment-70286788 #955]<br />
** RR will merge<br />
* [RR] What should copyright headers say? [https://github.com/VisTrails/VisTrails/pull/994#issuecomment-70173935 #994]<br />
** Let's keep everything in there: Utah/Poly/NYU<br />
<br />
=== January 14, 2015 ===<br />
'''Updates'''<br />
* [TE] Working on classifier<br />
* [RR] Scripting integration, work in progress<br />
<br />
'''Items to Discuss'''<br />
* [RR] Unified wrapping method discussion ([https://github.com/VisTrails/VisTrails/issues/991 #991])<br />
** Let's talk next week, [AM] and [DK] are not here<br />
<br />
=== January 7, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* make sure that we address critical issues, questions, and pending review branches in a timely manner<br />
* scripting support<br />
** [RR] no issues if we want to just keep annotations in the generated code to allow the link back to a workflow<br />
** [RR] can translate from workflow to script, working on script to workflow<br />
** will work for parameter value changes, structural changes require changes to the annotations<br />
** need to publish best practices here<br />
** would be cool to do looping in scripts (easier interface than with workflows)<br />
* notebook support (convert form notebook to workflow)<br />
** RR will sync with FC on this<br />
* Issue with console in built-from-scratch<br />
** [TC] iPython rearranged some of the completion stuff in 2.2 and 2.3<br />
** binary has old version of iPython -> 1.0.0, should we update?<br />
* [TC] automated wrapping of numpy and scipy<br />
** discovered a bunch of malformed documentation in numpy and scipy<br />
** has github repo for vistrails tools<br />
** example modules wrap a bunch of R stuff (not baked in, just how things are)<br />
** will be pushing wrapping logic up<br />
** port names forbidden (window and domain)<br />
** have an import hook to get from yaml directly to VisTrails Modules<br />
** should work for any python modules with well-formed numpy docstrings.<br />
* [Action] should make it clear in documentation that Constant now means serializable not that the value doesn't change (e.g. List)<br />
* [TC] might be interesting to try to build components of matplotlib and accumulate in figure (long-term project, but thinking about how this might work)<br />
* [TE] build and build scripts<br />
** completely automatic, buildbot<br />
** need to set the build machines for the environment we want for the binary<br />
** would virtualenv work here?<br />
** [TC] anaconda can pin versions, potential path to test different configurations<br />
** Q: upload nightly binary builds? A: makes sense, make sure they are well-labeled<br />
* sourceforge stats: e.g. http://sourceforge.net/projects/vistrails/files/vistrails/nightly/vistrails-src-nightly.tar.gz/stats/timeline?dates=2014-01-01+to+2015-01-07<br />
* package issues (see Remi's message)<br />
* [TE] Scope of tej<br />
** Support single ssh commands?<br />
** Queue can be used as a remote machine (crawler is using queue.call*)<br />
* SourceForge stats: http://sourceforge.net/projects/vistrails/files/vistrails/nightly/vistrails-src-nightly.tar.gz/stats/timeline?dates=2014-01-01+to+2015-01-07<br />
<br />
== Older meetings ==<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=11861Development2016-05-25T14:41:45Z<p>Remi: /* May 25, 2016 */</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2016 ==<br />
<br />
=== May 25, 2016 ===<br />
<br />
'''Updates'''<br />
* Mailing-list fixed<br />
* Getting stats from the "server news" mechanism (via [https://gist.github.com/remram44/7b955a0a2cfa9382f89702531d1fe6f8 parser])<br />
<br />
'''Items to Discuss'''<br />
* Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)<br />
<br />
=== May 18, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] mailing list still down - Julio working on it<br />
* [TE] Improved union port<br />
** New schema 1.0.5 adds portitem.union string attribute<br />
** Works with pipeline view, parameter exploration view, and mashup view<br />
** Should work well for matplotlib<br />
** Use with VTK ports?<br />
*** It uses [item_1,item_2] style ports usually with different tuple length.<br />
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]<br />
<br />
=== May 11, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?<br />
*** Shows single port for all ports with same sort_key<br />
*** Select specific type when adding/showing<br />
*** Only one type should be visible on the module at any time<br />
*** Making tuple parameters union types is probably too complicated<br />
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly) <br />
<br />
* [TE] Feature suggestion: Let connections connect to functions/parameters<br />
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters<br />
** Use separate views for ports and functions (like before)<br />
*** Allow rearranging of functions<br />
** Pros:<br />
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.<br />
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.<br />
**** Makes List module simpler<br />
*** Connect to parameter in tuple<br />
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds. <br />
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.<br />
** Cons:<br />
*** Interface more complicated and crowded<br />
*** More difficult for users to grasp?<br />
*** No immediate use case<br />
*** Makes no union type for tuple parameters a bigger problem.<br />
<br />
* [TE] Publish user survey?<br />
** No response from vistrails release yet<br />
** Not announced on vistrails-users yet.<br />
<br />
=== May 4, 2016 ===<br />
'''Updates'''<br />
* VisTrails 2.2.4 released<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
<br />
=== April 27, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Wrapping Bokeh<br />
** New method type "operation"<br />
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output<br />
**** Glyph sets function name and arguments as output<br />
***** Not serializable, but should always be computed together with the subject of the operation.<br />
**** Figure applies Glyph operation on itself<br />
*** Useful for matplotlib?<br />
**** matplotlib currently find actors by searching upstream?<br />
**** But this is more difficult since matplotlib is not function-based.<br />
** New "union" port type<br />
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value<br />
**** Uses only first value in tuple<br />
**** Bokeh have many properties that support (list, string, Float)<br />
*** Does not work with depth=1 types (e.g. Float of depth 1)<br />
**** Turn Depth 1 types with known type into lists?<br />
<br />
* [TE] New bugfix release<br />
** Finalize survey<br />
<br />
=== April 20, 2016 ===<br />
'''Updates'''<br />
* Merged upgrade fixes<br />
** Now has infinite loop test<br />
** loop test now correctly loops 5 times<br />
*** remap_module used wrong (current) package version when creating ModuleDescriptor<br />
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])<br />
<br />
'''Items to Discuss'''<br />
* New bugfix release<br />
** Merge usagestats<br />
** Need to finalize survey<br />
<br />
=== April 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?<br />
<br />
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])<br />
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]<br />
** Cherry-pick test to preserve source?<br />
** Need maxPipelineFixAttempts option?<br />
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.<br />
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?<br />
*** Would need to be moved into handle_invalid_pipeline<br />
<br />
=== April 6, 2016 ===<br />
'''Updates'''<br />
* [TE] Query View improvements<br />
** Done except for displaying "Version Info" and #1164<br />
* [RR] Working on finishing new interpreter (finally!)<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Try latest upgrade first, then previous.<br />
** If all fail, select latest upgrade?<br />
<br />
=== March 30, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Query View improvements<br />
** Unified do_version_switch and get_upgrades as validate_version()<br />
** Problems with query view using its own controller<br />
*** Fixed module info not being displayed for query results<br />
*** Still no version info for query results<br />
*** Still an issue with version query view not being updated<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== 2015 ==<br />
<br />
=== December 30, 2015 ===<br />
'''Updates'''<br />
* Bokeh package finished<br />
** Can output to Browser/IPython/Spreadsheet (Limited functionality in spreadsheet)<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Bokeh needs QWebEngine (Chromium) in Qt5 which requires PyQt5<br />
*** Many plotting libraries only work in browsers<br />
*** But Chromium may be a big and controversial dependency<br />
*** Or create a browser version of the spreadsheet?<br />
** Some users have requested Python 3<br />
** Not backwards compatible?<br />
** Target for VisTrails 3.0 with new interpreter?<br />
<br />
=== December 16, 2015 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Users list question<br />
* [TE] Wrapping Bokeh<br />
** Added bokeh property parser<br />
*** complements numpydoc parser<br />
*** Not all attributes are available, partially because I am using a pre-release version.<br />
** Wrapped modules<br />
*** bokeh.plotting finished (Backwards like matplotlib: Create plot then draw on it)<br />
*** bokeh.charts mostly done (Missing arguments)<br />
** QWebView mostly works (but no zooming or buttons)<br />
*** It would be better to use external browser or QWebEngineView<br />
** Added improvements to wrapper<br />
*** Generalized port translations (Color/Path to native types)<br />
*** Can use multiple docstring/property parsers simultaneously<br />
<br />
=== December 9, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Looking into wrapper<br />
** Will port TensorFlow<br />
** Doing a pass on the wrapper code & doc<br />
*** UX improvements<br />
** Factorize high-level docstring parsers into core.wrapper? (sphinx, numpy, google)<br />
** Problem with name vs module_name (fails sklearn tests because class_by_name() is broken)<br />
** Looking into compressing specs, lazy registration of modules<br />
<br />
=== December 2, 2015 ===<br />
'''Updates'''<br />
* [TE] Package wrapping<br />
** Wrapped most of numpy/scipy using __all__<br />
** Added class attribute and method access for classes (Can add to class or to separate inspector module)<br />
** TODO: Document wrapping procedure<br />
<br />
'''Items to Discuss'''<br />
<br />
=== November 25, 2015 ===<br />
'''Updates'''<br />
* [RR] New interpreter is coming along<br />
** Everything is a stream<br />
** Task system, eventually work-stealing parallelism<br />
** Streams no longer advance in lock-step<br />
*** Allows for constructions like filter, join, sorted-merge<br />
** depth>1 still to be tested, no current plan to allow this through the module interface<br />
*** Through looping though, can do streams of streams<br />
<br />
'''Items to Discuss'''<br />
* [TE] Package wrapping<br />
** Wrapped most numpy functions (no polynomials)<br />
** and half of scipy classes and functions<br />
** Added functions manually from documentation (no parseable lists?)<br />
** Focusing on data flow constructs (no property/method access for classes)<br />
*** Using spec diff (empty for now) and dynamic parser (stores spec in .vistrails/)<br />
** Python class/function wrapper now stable<br />
** Add google doc parser?<br />
*** Is there a parser available?<br />
<br />
=== November 18, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Package wrapping<br />
** Numpy class wrapping works<br />
*** Basic type is List, since most types are array_likes.<br />
*** TODO: More classes and functions<br />
*** Some operations are in-place and some aren't, and docstring not super clear<br />
*** [RR] It seems only methods mutate, and most have a numpy.xx function equivalent; just manually go through the methods and remove the mutating ones?<br />
** Class wrapper is modular<br />
*** Docstring parser and type string parser can be customized<br />
*** Classes can have optional attribute/method ports<br />
**** Inspectors and attribute/method modules can be created separately<br />
** Use PythonCalc as an example<br />
*** Can be re-implemented as a function with a parseable docstring<br />
*** [RR] PythonCalc is just an example, and it might actually make sense to get rid of it, or do a proper math package with scalar operations (as separate modules, no combobox)<br />
** [RR] TensorFlow to use wrapping as well<br />
*** Very simple (only types 'tensor' & 'variable')<br />
<br />
=== November 11, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Package wrapping<br />
** Wrapping numpy's classes using numpydoc (ndarray)<br />
*** Wrapping constructor arguments, attribute getters/setters as module ports<br />
**** Then how to access attributes afterwards?<br />
***** Use input value port to class modules?<br />
*** Wrap methods as modules? E.g., `ndarray.shape`.<br />
*** Function wrapping is almost a subset of class wrapping<br />
**** May be able to use the same parsing/execution methods<br />
** VTK's non-getter/setter methods could be wrapped as modules<br />
*** We could then remove extra logic in interpreter for keeping function order<br />
** Bokeh uses autogenerated docstrings<br />
*** We may be able to read the specification directly<br />
<br />
* [RR] [https://github.com/VisTrails/VisTrails/pull/1138 TensorFlow package]<br />
** Basic setup working, can execute the Mandelbrot example<br />
** Will autogenerate the operations<br />
<br />
=== November 4, 2015 ===<br />
'''Updates'''<br />
* [RR] Interpreter work requires a fix for [https://github.com/VisTrails/VisTrails/issues/1137 DB issue #1137]<br />
* [DK] Kitware's Resonant [http://www.kitware.com/source/home/post/173]<br />
** [https://github.com/girder/girder Girder]: data management system<br />
** [https://github.com/Kitware/romanesco Romanesco]: execution engine, uses [http://www.celeryproject.org/ Celery] for task management<br />
** Resonant Flow: web app for editing and executing workflows<br />
<br />
'''Items to Discuss'''<br />
* [TE] Added OSX Lion (10.7) VM on build machine using vagrant (seems ok with license?)<br />
** New builds works on Lion<br />
** Change the minimum requirement to 10.7?<br />
* [TE] Package wrapping<br />
* Added upgrade suggestions using 2 spec versions ([http://pastebin.com/30XUU8Gp example])<br />
** using name edit distance to find matches<br />
** Could use something better like port similarity for modules, and type similarity for ports<br />
** Should write python upgrade code?<br />
** Added spec to sklearn package<br />
*** Can now diff spec versions and keep the spec static<br />
*** There will be problems with downgrades and package versions<br />
<br />
=== October 29, 2015 ===<br />
'''Updates'''<br />
* [TE] nightly-build-mac fixed (Needed to approve xcode license)<br />
<br />
'''Items to Discuss'''<br />
* [TE] matplotlib wrapper<br />
** ported to general spec<br />
** Diff can be used on general spec<br />
*** Only needed minor changes<br />
*** Fixed indexing bug that corrupted specs<br />
*** Can now move diff tools to core/wrapper<br />
** Add new plots?<br />
*** New plots in mpl 1.3: eventplot<br />
*** New plots in mpl 1.4: angle_spectrum, magnitude_spectrum, phase_spectrum', violinplot<br />
*** No new plots in mpl 1.5 (from looking at boilerplate.py)<br />
<br />
=== October 21, 2015 ===<br />
'''Updates'''<br />
* [RR] Alexis has arrived, will be working with [RR] on a more efficient interpreter<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1133 New matplotlib package]<br />
** Supporting multiple package versions<br />
*** Check which version can be loaded [https://github.com/VisTrails/VisTrails/pull/1135 #1135]<br />
*** We cannot show version requirements in the list of packages because old packages are loaded when importing the codepath<br />
** Visual diff does not work well, but we can use the spec differ to see differences.<br />
** Unifying vtk and matplotlib wrapping specs (WIP)<br />
*** And porting matplotlibs spec differ to work on the general spec<br />
*** Spec differ can then be used on vtk and other packages in the future <br />
*** Will enable automatic upgrade generation<br />
* [RR] Rework interpreter<br />
** restore abstraction between interpreter/module code<br />
** build looping, streaming, caching into the interpreter<br />
** rework cache<br />
** look into parallelism once basic functionality is there<br />
<br />
=== October 14, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1133 New matplotlib package] (adding numpydoc parser)<br />
<br />
=== October 7, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://github.com/VisTrails/VisTrails/tree/fix-matplotlib-parser New matplotlib parser] (adding numpydoc parser)<br />
** Added numpydoc attribute parser for plots.<br />
*** TODO: Need better port spec reconciliation with call signature parser.<br />
*** We can create a general numpydoc parser, but attribute types are unique to matplotlib.<br />
** Needs package versioning<br />
*** New spec will not support matplotlib < v1.4 due to changed path to axes classes.<br />
*** Load spec version corresponding to installed matplotlib version?<br />
*** Need version downgrades?<br />
<br />
=== September 30, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Updating Matplotlib parser<br />
** Matplotlib docstring parser fails on numpy docstrings<br />
** I have [https://github.com/VisTrails/VisTrails/commit/eae8b0c16260810587e47c41cab53552cbb3cf0a added basic numpydoc parsing]<br />
*** Only used by a few docstrings so far<br />
*** Parsing uses many sources (class tables, signatures, ACCEPTS, method docstrings, definition parsing).<br />
** Caswell said they were thinking about moving to traitlets, but this is not ready yet<br />
<br />
=== September 23, 2015 ===<br />
'''Updates'''<br />
* [TE] VisTrails 2.2.3 released<br />
** Also have pushed to PyPI, binstar, etc.<br />
** Sourceforge vs. GitHub<br />
*** should be able to host releases on GitHub<br />
*** nightly binaries pushed to sf each night<br />
*** old binaries?<br />
* [TE] Re-wrapping MatPlotlib<br />
** Keep static generation - Docstrings are brittle beween matplotlib versions<br />
** Keep generating the executable classes - but create functions, not vistrails Modules <br />
*** Why is there so much patching?<br />
** Re-implemented Alternate PortSpec for InputPortSpec<br />
*** It will now inherit specs from parent (No need to reimplement)<br />
** Module upgrades?<br />
<br />
'''Items to Discuss'''<br />
<br />
=== September 16, 2015 ===<br />
'''Updates'''<br />
* [TE] Job Monitor tests and documentation done<br />
** OK to leave document package in packages directory?<br />
<br />
'''Items to Discuss'''<br />
* [TE] Library wrapping: How to do code patching? [https://docs.google.com/drawings/d/1Sh3eOXot6v2y880jHKQrqXqAwr-0aoqYligg3z8G8MY/edit?usp=sharing Wrapping diagram]<br />
** Insert code into module template (Old VTK code)<br />
** Patch the library that is called (New VTK code)<br />
** Somehow store code in spec and apply when executed?<br />
*** Executed code needs at a minimum access to inputs, outputs, current class<br />
*** Use standard input/output dict and obj reference that the code operates on?<br />
*** Can this be done while keeping the execution abstract?<br />
<br />
* Do new release now?<br />
<br />
=== September 9, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* New release?<br />
** [RR] wants reprounzip<br />
** tej changes & doc<br />
** job monitor & jobmixin fixes<br />
** mongodb<br />
** reprounzip<br />
** warning: MplFigure type is int<br />
** tabledata: add Reshape, DictoToTable & ListToTable, fix ListToTable with numpy arrays<br />
** PythonSource can have same name for an input & output<br />
** don't show spreadsheet at exit<br />
<br />
=== September 2, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Limit autosaves? [https://github.com/VisTrails/VisTrails/pull/1126 #1126]<br />
<br />
=== August 26, 2015 ===<br />
'''Updates'''<br />
* [TE] Added Job support to Parameter Explorations (Requested by Colin), and Mashups<br />
** specify job ids, need to specify different ids for parameter explorations since they have the same version id<br />
** how to deal with parameters passed in on command line<br />
* [General] provenance: should be creating a new version when we execute workflow with changed parameters?<br />
** currently, custom_params annotation stores this in provenance currently<br />
** [RR] API doesn't record provenance if passing in parameters: http://git.io/vsAA5<br />
'''Items to Discuss'''<br />
<br />
=== August 19, 2015 ===<br />
'''Updates'''<br />
* [RR] Improvements to tej, users' guide entry & example [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
* [RR] Internal docs [https://github.com/VisTrails/VisTrails/pull/1116 #1116]<br />
** On ReadTheDocs: [http://vistrails.readthedocs.org/ vistrails.readthedocs.org]<br />
* [TE] Working on Job Monitor documentation<br />
<br />
'''Items to Discuss'''<br />
* [RR] JobMixin and JobMonitor: stable now?<br />
** Definitely needs more tests<br />
* [TE] Stop testing VisTrails 2.0?<br />
<br />
=== August 12, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Internal docs [https://github.com/VisTrails/VisTrails/pull/1116 #1116]<br />
** On ReadTheDocs: [http://vistrails.readthedocs.org/ vistrails.readthedocs.org]<br />
* [RR] JobMixin and JobMonitor: stable now?<br />
** Definitely needs more tests<br />
* [RR] Improvements to tej, users' guide entry & example [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
* [RR] Build broken on Travis; because of IPython 4 released today? ([https://travis-ci.org/VisTrails/VisTrails/builds/75279795 build 992]; [https://github.com/VisTrails/VisTrails/issues/1123 #1123])<br />
<br />
=== August 5, 2015 ===<br />
'''Updates'''<br />
* [TE] Fixes to Jobs<br />
** Could not delete jobs<br />
** Could not run job in group<br />
** Job not reset when calling ModuleSuspended<br />
** Added deleting job from context menu<br />
* [RR] Writing documentation for everything [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
<br />
'''Items to Discuss'''<br />
<br />
=== July 29, 2015 ===<br />
'''Updates'''<br />
* RR still looking into new interpreter thing<br />
** Goal is to take out scheduling logic from Module so it can be split in multiple processes, and so that smarter strategies can be added in time<br />
** This means some work on packages<br />
** Spreadsheet can live in kernel process? Still some UI stuff that will take work (changing configuration, persistent archive's viewer, ...)<br />
* [TE] Problems running examples [https://github.com/VisTrails/VisTrails/pull/1111 #1111]<br />
** Testing of more examples requires additional packages on the test machines.<br />
** Fixed faulty line-ending in PythonSource:s failing on Python 2.6.<br />
** Test suite now testing SUDSWebServices (If web service is down, test suite will fail)<br />
** preferences.py test failed reloading 'dialogs' package, switched to using 'URL'.<br />
<br />
'''Items to Discuss'''<br />
<br />
=== July 22, 2015 ===<br />
'''Updates'''<br />
* [TE] mailing lists back online<br />
* [TE] Working on [https://github.com/VisTrails/VisTrails/issues/1107 #1107]<br />
* [RR] Working on ReproZip package <br />
<br />
'''Items to Discuss'''<br />
* [RR] MongoDB package [https://github.com/VisTrails/VisTrails/pull/1106 #1106]<br />
* [RR] Example for tej docs? [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
<br />
=== July 15, 2015 ===<br />
'''Updates'''<br />
* 2.2.2 released<br />
<br />
'''Items to Discuss'''<br />
* RR is considering executing everything in an IPython kernel (i.e. separate Python interpreter, like the one spawned for a notebook)<br />
** It's a separate process that we can restart without restarting the app/gui<br />
** We can isolate the execution environment (e.g. for the server)<br />
** We can run the whole thing remotely (if your data is elsewhere, just run VisTrails locally and the pipeline elsewhere)<br />
** Pipeline execution no longer makes the interface hang, it just makes the kernel hang (but that's fine)<br />
** We can use notebooks as modules (probably way nicer than the PythonSource module)<br />
** We can run multiple kernels so long as the ports carry things that are serializable<br />
*** meaning we can put the multithreaded-interpreter without all the hacky parts it has now<br />
*** we can run IPython kernels in all the languages IPython supports, [https://github.com/ipython/ipython/wiki/IPython-kernels-for-other-languages currently 46]<br />
<br />
=== July 8, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Buildbot github hook not working after IP address change<br />
* [TE] No reference to VistrailsApplication ([https://github.com/VisTrails/VisTrails/issues/1103 #1103])<br />
** [RR] Added PR [https://github.com/VisTrails/VisTrails/pull/1104 #1104]<br />
* [TE] Reopening VT file after saving with bundled subworkflow won't offer subworkflow upgrade ([https://github.com/VisTrails/VisTrails/issues/1102 #1102])<br />
** allow manual delete to fix right now<br />
** fix this on top of the use-uuid branch<br />
* [TE] Release VisTrails 2.2.1? ([https://github.com/VisTrails/VisTrails/blob/v2.2/CHANGELOG CHANGELOG])<br />
** [https://github.com/VisTrails/VisTrails/pull/1093 output-modules-use-global-dir]<br />
** [https://github.com/VisTrails/VisTrails/milestones/version%202.2 Current issues for 2.2]<br />
<br />
=== July 1, 2015 ===<br />
'''Updates'''<br />
* [TE] PROV fixed<br />
* [TE] Working on subworkflow issues<br />
* [RR] Considering reworking the controller (log vs exception problem, retained upgrades causing interferences) and also the interpreter (IPython?)<br />
<br />
'''Items to Discuss'''<br />
<br />
=== June 24, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* UV-CDAT<br />
* [TE] Can a cyclic workflow be valid? ([https://github.com/VisTrails/VisTrails/issues/1097 #1097])<br />
** focus on disabling the ability to create cyclic pipelines because more things break than just this with a cyclic pipeline<br />
* [RR] Relative paths ([https://github.com/VisTrails/VisTrails/pull/1057 #1057])<br />
** This interacts with the new bundle; how do handle packing files inside the VT bundle?<br />
<br />
=== June 17, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] current_version and reusing existing upgrades are broken ([https://github.com/VisTrails/VisTrails/issues/1095 ticket #1095])<br />
** Could be that export to PROV is using an unflushed upgrade pipeline?<br />
** current_version would then be correct.<br />
** It may work to flush the actions before exporting?<br />
<br />
=== June 10, 2015 ===<br />
'''Updates'''<br />
* DAT: fixed VTK issue on Linux and Mac<br />
* Still crashes on Windows. Need help! Reminder: this works in the VisTrails spreadsheet (QCellPresenter), although no widget get changed there during a drag<br />
** Is it a VTK bug?<br />
** Is it simply impossible to change widgets during the drag, should we do it a different way?<br />
** Did I miss something that is done in VisTrails but somehow not in DAT?<br />
** Low prio, UV-CDAT doesn't run on Windows anyway<br />
<br />
'''Items to Discuss'''<br />
* [TE] Executable Paper ([https://github.com/VisTrails/VisTrails/issues/1086 ticket #1088] [https://github.com/VisTrails/VisTrails/pull/1088 Updated pull request])<br />
* Single instance code and batch mode<br />
<br />
=== June 3, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Executable Paper ([https://github.com/VisTrails/VisTrails/issues/1086 ticket #1088] [https://github.com/VisTrails/VisTrails/pull/1088 pr])<br />
** Requires fixes to command line parameters, Output modules, and batch mode<br />
** How to test this<br />
** Updated missing/outdated flags<br />
** Fixed view issues when generating graphs<br />
* [TE] batch mode<br />
** SpreadsheetOutput not enabled in batch mode. Should we check is_running_gui instead?<br />
** Other instance setting flags from caller<br />
** Is graphsAsPdf replacing spreadsheetDumpPdf?<br />
** graphsAsPdf true by Default?<br />
** Batch mode executing by default (Not needed when generating graphs)<br />
** Re-added workflowInfo as withWorkflowInfo for writing graph and xml workflow<br />
** [DK] batch mode should be outputting to files or stdout, shouldn't always trigger SpreadsheetMode<br />
** execute flag, maybe make execution the default and allow a "--no-execute" if you only want to capture graphs, for example<br />
<br />
=== May 27, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] API question<br />
** http://lists.vistrails.org/pipermail/vistrails-users/2015-May/000357.html<br />
* Status of DAT<br />
** Can add VCS plot to DAT but need to work on configuration windows<br />
** Documentation for graphics templates for UV-CDAT/vcs?<br />
* Qt support?<br />
<br />
=== May 20, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Reviving the DAT, integrating scripting & porting UV-CDAT!<br />
** On [https://github.com/VisTrails/DAT GitHub] ([https://github.com/VisTrails/DAT/issues issues])<br />
** Merging 2 years of development taking longer than expected, but getting there. The plan is to get the patches in VisTrails and never fork again, we never want to get in UV-CDAT's situation (and don't need to).<br />
** VTK cell works fine on Linux but there was flickering on Mac & Windows before; still issues on Mac (Windows status unknown)<br />
** Can get a VCS plot soon (but will need VTK cell fix)<br />
** Integrate in UV-CDAT's build system (so we have cdms, VCS, ...) -> RR can do this, low priority<br />
** How do we integrate scripting?<br />
*** We want to be able to seamlessly make changes to a plot by changing Python code<br />
*** Define new plots by entering Python code without writing modules/packages?<br />
* [RR] UV-CDAT [https://github.com/UV-CDAT/uvcdat/issues/1275 needs] [https://github.com/VisTrails/VisTrails/pull/1073 #1073], please take a look<br />
* [RR] Subworkflow issues<br />
** [https://github.com/VisTrails/VisTrails/issues/1065 #1065]: missing attribute (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/1066 #1066]: groups in subworkflows were ignored when finding dependencies (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/725 #725]: "upgrade subworkflow" button not popping up (fixed); also, namespace issue<br />
** [https://github.com/VisTrails/VisTrails/issues/1071 #1071]: subworkflows don't go through upgrades<br />
* [RR] [https://github.com/VisTrails/VisTrails/issues/1074 #1074]: can't load and edit a single pipeline if upgrade happen<br />
<br />
=== May 13, 2015 ===<br />
'''Updates'''<br />
* [RR] Test skipping whitelist ([https://github.com/VisTrails/VisTrails/pull/1069 #1069]) -- low priority<br />
* [RR] Custom matplotlib modules can't be compatible with both 2.1 and 2.2 ([https://github.com/VisTrails/VisTrails/issues/1067 #1067]); should be fixed for ALPS ([https://github.com/VisTrails/VisTrails/issues/1070 #1070])<br />
<br />
'''Items to Discuss'''<br />
* [RR] UV-CDAT [https://github.com/UV-CDAT/uvcdat/issues/1275 needs] [https://github.com/VisTrails/VisTrails/pull/1073 #1073], please take a look<br />
* [RR] Subworkflow issues<br />
** [https://github.com/VisTrails/VisTrails/issues/1065 #1065]: missing attribute (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/1066 #1066]: groups in subworkflows were ignored when finding dependencies (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/725 #725]: "upgrade subworkflow" button not popping up (fixed); also, namespace issue<br />
** [https://github.com/VisTrails/VisTrails/issues/1071 #1071]: subworkflows don't go through upgrades<br />
* [RR] [https://github.com/VisTrails/VisTrails/issues/1074 #1074]: can't load and edit a single pipeline if upgrade happen<br />
<br />
=== May 7, 2015 ===<br />
'''Updates'''<br />
* [RR] Export/import workflow to Python working!<br />
<br />
'''Items to Discuss'''<br />
* [Claudio] UV-CDAT<br />
** The UV-CDAT project is the biggest user base of VisTrails<br />
** VisTrails package management provides a lot of friction towards people plugging in their code<br />
** Need to make it easy to integrate your random Python scripts in the system without having to deal with all the boilerplate, at least in the first step<br />
** [RR] argues that modules are still good; UV-CDAT shouldn't move towards a purely script-based backend<br />
** [RR] export/import with Python could reduce a lot of that friction by allowing 1) to edit workflow as Python 2) to open up boxes automatically if needed code doesn't match actual modules<br />
** ...<br />
<br />
=== April 29, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] <strike>Abstractions</strike> subworkflows status ([https://github.com/VisTrails/VisTrails/labels/subworkflows tickets])<br />
* [RR] matplotlib compatibility (2.1 & 2.2), [https://github.com/VisTrails/VisTrails/issues/1067 #1067]<br />
** RR to try and fix ALPS matplotlib modules<br />
<br />
=== April 22, 2015 ===<br />
'''Updates'''<br />
* 2.2.0 has been released!<br />
* Windows issue (via email from Ryan)<br />
** issue with manifest file (may be a new file in VTK6?)<br />
** Tommy has regenerated new Windows builds<br />
* Binaries, pypi, and conda released<br />
* [RR] Export as script<br />
* Python sources using VTK need to switch to SetInputData (users should be aware of this)<br />
<br />
=== April 15, 2015 ===<br />
'''Updates'''<br />
* Ready for 2.2.0 (apart from binary/deps issues)<br />
** Missing some libs (scikit-learn, tej, tdparser, SQLAlchemy+connectors<br />
** Windows: runvistrails.py is no longer used, so the PATH is wrong<br />
** Windows: pip is broken, but it probably wouldn't work anyway because of permission issues (disable this?)<br />
* Queries, upgrades and getPipeline() usage ([https://github.com/VisTrails/VisTrails/issues/1054 #1054])<br />
** Getting a pipeline with getPipeline() is not safe: it might return an invalid pipeline<br />
** This is used in many places throughout the code, like queries<br />
** Upgrading would require going through the controller, but that creates new actions<br />
* [TC] Avoid copying a module's output if it's used as input by exactly one downstream module ([https://github.com/VisTrails/VisTrails/issues/1060 #1060]) (useful for big numpy arrays you can update in-place)<br />
<br />
=== April 8, 2015 ===<br />
'''Updates'''<br />
* [DK] Merged RR's changes for output modules (1012 and 1013)<br />
** RR will merge remaining changes, then create v2.2 branch!<br />
<br />
'''Items to Discuss'''<br />
* BNL need numpy array to VTK image<br />
** Looks like VTK has some helpers for this<br />
** We will help if issues arise<br />
** Will contribute back to VisTrails package<br />
* Upgrade issue: [https://github.com/VisTrails/VisTrails/issues/1017 #1017]<br />
** Automatic upgrades should happen between versions of provided upgrades<br />
** Our existing upgrades work around this so it doesn't need to be 2.2.0<br />
* Corner-case VTK modules<br />
** No longer need VTKCell input port, so don't interfere with registry and API anymore<br />
** Still work weirdly, people probably shouldn't use them<br />
** But we have lots of clunky modules since we wrap the whole of VTK; some people might rely on this and know how to use them, let's keep them anyway<br />
** Ready for 2.2.0<br />
<br />
=== April 1, 2015 ===<br />
'''Updates'''<br />
* [RR] UV-CDAT: bugfixing for 2.2, long-term plans: implement scripting import/export in VisTrails, port to UV-CDAT<br />
* Possibly, try to move to regular VisTrails to use new features<br />
<br />
'''Items to Discuss'''<br />
* 2.2 release: nothing much is pending anymore, release next week?<br />
** [TE] vtkExporter classes [https://github.com/VisTrails/VisTrails/issues/1032 #1032]<br />
** [RR] Hiding upgrades in version tree might make it (or might be 2.2.1 so we can test it out) [https://github.com/VisTrails/VisTrails/issues/949 #949]<br />
** Output modules changes to go in<br />
* add note to documentation about order of parameters in VTK<br />
* add issue about exporter upgrades if not already there<br />
<br />
=== March 25, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* 2.2 release<br />
** Checklist on Github: https://github.com/VisTrails/VisTrails/wiki/2.2-Release-Checklist<br />
** Review output modules<br />
** RR has a couple more issues to fix<br />
** Ready to go -- sign app package for OSX? [https://github.com/VisTrails/VisTrails/issues/984 #984] We need access to the Apple Membership team<br />
<br />
=== March 18, 2015 ===<br />
'''Updates'''<br />
* [TE] VTK6 works<br />
<br />
'''Items to Discuss'''<br />
* [RR] Release v2.1.5 with backported tabledata?<br />
** MTA example needs updated tabledata (for JoinTables)<br />
** Google Maps package still not available<br />
* [RR] Work torward v2.2.0?<br />
** changes:<br />
*** new persistence<br />
*** API changes<br />
*** output module changes (upgrades?), maintain cells but try to upgrade<br />
*** not wrapping stuff<br />
*** VTK6? yes<br />
*** JobSubmission stuff?<br />
*** relabeling for upgrades [https://github.com/VisTrails/VisTrails/issues/949 #949]<br />
** makes sense, needs the tree view code to be updated, check selection<br />
** See [https://github.com/VisTrails/VisTrails/wiki/2.2-Release-Checklist 2.2 checklist]<br />
* Discussion of [https://github.com/VisTrails/VisTrails/issues/1016 #1016]<br />
** plumbing between outputs and output modes, how to define a mode that works for many outputs without writing for each output?<br />
<br />
=== March 11, 2015 ===<br />
'''Updates'''<br />
* [RR] persistent_archive done; merge? ([https://github.com/VisTrails/VisTrails/pull/755 #755])<br />
** note about the focus events for widgets<br />
** TE be aware of file_archive for future binaries that include persistent_archive<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/998 New VTK package] finished<br />
<br />
'''Items to Discuss'''<br />
* [RR] Unmark UV-CDAT/VisTrails as a fork of VisTrails/VisTrails? (see last week; decision needed)<br />
** RR email to JF about this<br />
** yes; email sent to Github<br />
* [RR] Switching order of output ports ([https://github.com/VisTrails/VisTrails/pull/1006 #1006])<br />
** added port specs are sorted at a separate spot (Module.*_port_specs properties) than those in the registry (which are sorted in the registry), but those two lists are just combined without respect to sort keys<br />
** need to determine whether the two lists should be merged or remain distinct<br />
** should make sure that order of input ports and output ports makes parallel connections for things with same order<br />
** DK suggests breaking backward compatibility here: workflows still run, can fix easily if a problem in an existing package.<br />
* [RR] Question about output modes ([https://github.com/VisTrails/VisTrails/pull/1007 #1007]), how to integrate in API ([https://github.com/VisTrails/VisTrails/pull/24 #24])<br />
** Should ImageFileMode be removed? ("image" is not a mode, "file" is)<br />
** ImageOutput missing?<br />
** Feel free to change how formats works<br />
* [TE] Test suite segfaults on Fedora 17 virtualbox. Install newer version? (Support ended in 2013)<br />
<br />
=== March 4, 2015 ===<br />
'''Updates'''<br />
* [TE] VTK wrapper<br />
** Works on VTK 5.10<br />
** Still need to test VTK 6<br />
** New general wrappers for python functions and classes <br />
<br />
'''Items to Discuss'''<br />
* [RR] Unmark UV-CDAT/VisTrails as a fork of VisTrails/VisTrails?<br />
** When [http://i.imgur.com/UTY2qfg.png filing a pull request] from UV-CDAT/VisTrails, VisTrails/VisTrails is selected by default<br />
** People keep forgetting to change the default ([https://github.com/VisTrails/VisTrails/pull/956 #956] [https://github.com/VisTrails/VisTrails/pull/968 #968] [https://github.com/VisTrails/VisTrails/pull/999 #999] [https://github.com/VisTrails/VisTrails/pull/1000 #1000] [https://github.com/VisTrails/VisTrails/pull/1003 #1003] [https://github.com/VisTrails/VisTrails/pull/1004 #1004] [https://github.com/VisTrails/VisTrails/pull/1005 #1005])<br />
** Only way to change that is to not have it marked as a fork of VisTrails/VisTrails<br />
** Github staff [https://gist.github.com/remram44/b364c24ef1e462ce6e2f can make that change for us]; should we do it?<br />
** Juliana: comments on this? visibility vs. convenience/annoyance for developers<br />
* [RR] What about [https://github.com/VisTrails/VisTrails/issues/991 #991]?<br />
<br />
=== February 25, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] vtkviewcell for infovis support, can we unify with VTKCell?<br />
** need to test this<br />
* [TE] vtk wrapping<br />
** Mostly finished<br />
** VTK 5.10 produce incorrect results with old wrapping<br />
*** Old wrapper is based mostly on VTK 4<br />
*** Most vtk_examples affected<br />
*** [http://www.vtk.org/Wiki/VTK/VTK_6_Migration/Replacement_of_SetInput]<br />
*** should be able to upgrade from SetInput to SetInputData (need to drop GetOutput and replace with self ports)<br />
*** can we change vtkInstance to just return self and not wrap things<br />
** terminator example not working under 5.8?<br />
* How does VTK wrapping fit into general wrapping framework?<br />
* [RR] new persistence package<br />
<br />
=== February 18, 2015 ===<br />
'''Updates'''<br />
* [RR] New VisTrails API and IPython integration ([https://github.com/VisTrails/VisTrails/pull/24 #24])<br />
<br />
'''Items to Discuss'''<br />
* [TE] VTK wrapping<br />
** [https://gist.github.com/rexissimus/16b74693545b5e7c1035 Benchmarking vtk package]<br />
*** Old: 24.7 seconds <br />
*** New: 10.5 seconds (Except first time that adds 8 sec)<br />
**** The parsing that calls is_abstract (that tries to instanciate all vtk classes) is now only run the first time.<br />
**** get_items_from_sigstring takes 2 seconds, maybe we can use a lookup dict for already computed sigstrings?<br />
** Now using a general python function wrapper<br />
*** VTK classes are wrapped into python function that does not depend on vistrails<br />
*** VTK functions can be executed without vistrails<br />
*** The spec maps functions into vistrails modules, but can also describe wrapping <br />
*** A general python function wrapper that supports <br />
**** kwarg inputs<br />
**** single, list, dict outputs<br />
**** callback for progress reporting<br />
**** temporary file generator for using FilePool<br />
**** optional output generation<br />
*** Creating specs:<br />
***** Create spec by hand<br />
***** Auto-create spec outline (TODO) and manually finish it<br />
***** Dynamically create spec (VTK)<br />
***** Implement documentation wrappers (Can use scikitlearn wrapper to wrap numpydoc) (TODO)<br />
***** Classes as bad functions needs to be wrapped in new functions before they are wrapped. This is different for each package.<br />
****** Classes is hard: Like VTK, and matplotlib. Scikit-learn does still not wrap classes<br />
***** Spec diffing and patching could be done using code from matplotlib.<br />
** Still needs upgrades from old VTK package<br />
*** Is it possible to dynamically wrap functions, e.g, you see a SetFunc and just remove the 'Set' prefix. Or do you need to create a complete mapping?<br />
<br />
=== February 11, 2015 ===<br />
'''Updates'''<br />
* Update from Friday's meeting<br />
** discussed VisTrails internals<br />
** discussed wrapping<br />
*** xml discussion, hard to modify because tied to db code<br />
*** TE has made it possible to add the schema-defined attributes to the intermediate representation<br />
*** higher-level operations on the port specs<br />
** make sure the simple case works<br />
*** [JF] take a simple package with documentation and figure out what the base case for wrapping is<br />
<br />
'''Items to Discuss'''<br />
* [TE] VTK wrapping<br />
** Dynamic loading works<br />
** Reading XML is fast enough, but [https://gist.github.com/rexissimus/8272d3c776ccfa115d82 unserializing data is slow]<br />
** Working on patterns for patching<br />
** Matplotlib has many advanced patterns like argument ordering, nested arguments, alternateSpecs, output types.<br />
*** Having all this in a general wrapper might confuse users?<br />
** [RR] Delay module (except for identifiers) until you need it---e.g. don't deal with port specs, etc. until necessary<br />
* Scripting Support [https://github.com/VisTrails/VisTrails/pull/950 #950]<br />
** [RR] Issue with getting code from modules<br />
** Design a simple solution<br />
** [JF] Couldn't you use modules as black boxes without conversion, just to call into modules/subworkflows easily from e.g. IPython?<br />
*** [RR] This is a job for the API, and a very separate use case. See [https://github.com/VisTrails/VisTrails/pull/24 #24]<br />
<br />
=== February 4, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* Wrapping<br />
** Format to use? Currently XML (like current matplotlib)<br />
*** JSON and YAML have simple "to python dictionary" methods<br />
*** But don't stream<br />
*** YAML a lot easier for humans<br />
** [DK] vtk-new-package also changes parameter names, creates enumerations<br />
*** intermediate schema needs to be extensible<br />
*** packages will want to store there specific infos for compute() method generation<br />
*** also might have specs-altering info, like matplotlib's alternateSpec<br />
** representation to code , registry already has schema for some aspects<br />
** [RR] We might want to see if Module subclasses can be created lazily<br />
*** no need to create all the classes just to register them in the registry and never actually use most of them<br />
*** future effort<br />
* [RR] Where should VisTrails packages live?<br />
** tej installs as 'vistrailspkg.tej', TE installed it as 'userpackages.tej'<br />
** Currently, standard packages are 'vistrails.packages.', user packages are 'userpackages.' and packages loaded through pkg_resources might be anything<br />
** [RR] Use 'vistrailspkg.' everywhere?<br />
** Long-term effort to simplify package distribution/installation (and have VisTrails get them automatically?)<br />
<br />
=== January 28, 2015 ===<br />
'''Updates'''<br />
* T. Caswell to come visit on Fri 6 to discuss wrapping work<br />
<br />
'''Items to Discuss'''<br />
* [TE] New VTK wrapping<br />
** Current code by DK seems a good deal faster<br />
** Generates XML that can be patched/tweaked, generates Python code from it, VisTrails only loads generated Python code<br />
** RR would rather have VisTrails load intermediate representation (= XML) directly, wants to make sure this is not slower<br />
** The goal is to turn the intermediate step into something generic that would be used for every wrapped package (vtk, numpy, matplotlib, sklearn, java) instead of each having its own<br />
** TC has its own code at [https://github.com/Nikea/VTTools/blob/master/vttools/wrap_lib.py github:VTTools] which parses numpy docstrings and generates modules, doesn't yet handle classes or persist anything<br />
* Web crawler<br />
** Right now, TE starts jobs for "start crawler", "stop crawler", "install classifier"<br />
** RR would rather have the crawling be a job as far as VisTrails and tej are concerned<br />
** The whole thing would be one pipeline: load examples, train classifier, start crawler [check for job, kill previous one, upload model, start processes], get snapshot, visualize<br />
*** Need some support in tej and job submission system: long-running jobs, stop a job (wait for it to finish?), restart a job even though results are cached<br />
<br />
=== January 21, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Unified wrapping method discussion [https://github.com/VisTrails/VisTrails/issues/991 #991]<br />
** TE to work on reusable method with intermediate representation, starting with VTK<br />
* [RR] Examples for scikit-learn: JF has an old example using Weka with parameter exploration (not currently in source tree)<br />
** AM's examples are enough<br />
* [AM] scikit-learn package is done, merge it in? [https://github.com/VisTrails/VisTrails/pull/955#issuecomment-70286788 #955]<br />
** RR will merge<br />
* [RR] What should copyright headers say? [https://github.com/VisTrails/VisTrails/pull/994#issuecomment-70173935 #994]<br />
** Let's keep everything in there: Utah/Poly/NYU<br />
<br />
=== January 14, 2015 ===<br />
'''Updates'''<br />
* [TE] Working on classifier<br />
* [RR] Scripting integration, work in progress<br />
<br />
'''Items to Discuss'''<br />
* [RR] Unified wrapping method discussion ([https://github.com/VisTrails/VisTrails/issues/991 #991])<br />
** Let's talk next week, [AM] and [DK] are not here<br />
<br />
=== January 7, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* make sure that we address critical issues, questions, and pending review branches in a timely manner<br />
* scripting support<br />
** [RR] no issues if we want to just keep annotations in the generated code to allow the link back to a workflow<br />
** [RR] can translate from workflow to script, working on script to workflow<br />
** will work for parameter value changes, structural changes require changes to the annotations<br />
** need to publish best practices here<br />
** would be cool to do looping in scripts (easier interface than with workflows)<br />
* notebook support (convert form notebook to workflow)<br />
** RR will sync with FC on this<br />
* Issue with console in built-from-scratch<br />
** [TC] iPython rearranged some of the completion stuff in 2.2 and 2.3<br />
** binary has old version of iPython -> 1.0.0, should we update?<br />
* [TC] automated wrapping of numpy and scipy<br />
** discovered a bunch of malformed documentation in numpy and scipy<br />
** has github repo for vistrails tools<br />
** example modules wrap a bunch of R stuff (not baked in, just how things are)<br />
** will be pushing wrapping logic up<br />
** port names forbidden (window and domain)<br />
** have an import hook to get from yaml directly to VisTrails Modules<br />
** should work for any python modules with well-formed numpy docstrings.<br />
* [Action] should make it clear in documentation that Constant now means serializable not that the value doesn't change (e.g. List)<br />
* [TC] might be interesting to try to build components of matplotlib and accumulate in figure (long-term project, but thinking about how this might work)<br />
* [TE] build and build scripts<br />
** completely automatic, buildbot<br />
** need to set the build machines for the environment we want for the binary<br />
** would virtualenv work here?<br />
** [TC] anaconda can pin versions, potential path to test different configurations<br />
** Q: upload nightly binary builds? A: makes sense, make sure they are well-labeled<br />
* sourceforge stats: e.g. http://sourceforge.net/projects/vistrails/files/vistrails/nightly/vistrails-src-nightly.tar.gz/stats/timeline?dates=2014-01-01+to+2015-01-07<br />
* package issues (see Remi's message)<br />
* [TE] Scope of tej<br />
** Support single ssh commands?<br />
** Queue can be used as a remote machine (crawler is using queue.call*)<br />
* SourceForge stats: http://sourceforge.net/projects/vistrails/files/vistrails/nightly/vistrails-src-nightly.tar.gz/stats/timeline?dates=2014-01-01+to+2015-01-07<br />
<br />
== Older meetings ==<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=11860Development2016-05-25T14:38:34Z<p>Remi: /* May 25, 2016 */</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2016 ==<br />
<br />
=== May 25, 2016 ===<br />
<br />
'''Updates'''<br />
* Mailing-list fixed<br />
* Getting stats from the "server news" mechanism (via [https://gist.github.com/remram44/7b955a0a2cfa9382f89702531d1fe6f8 parser])<br />
<br />
'''Items to Discuss'''<br />
<br />
=== May 18, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] mailing list still down - Julio working on it<br />
* [TE] Improved union port<br />
** New schema 1.0.5 adds portitem.union string attribute<br />
** Works with pipeline view, parameter exploration view, and mashup view<br />
** Should work well for matplotlib<br />
** Use with VTK ports?<br />
*** It uses [item_1,item_2] style ports usually with different tuple length.<br />
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]<br />
<br />
=== May 11, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?<br />
*** Shows single port for all ports with same sort_key<br />
*** Select specific type when adding/showing<br />
*** Only one type should be visible on the module at any time<br />
*** Making tuple parameters union types is probably too complicated<br />
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly) <br />
<br />
* [TE] Feature suggestion: Let connections connect to functions/parameters<br />
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters<br />
** Use separate views for ports and functions (like before)<br />
*** Allow rearranging of functions<br />
** Pros:<br />
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.<br />
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.<br />
**** Makes List module simpler<br />
*** Connect to parameter in tuple<br />
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds. <br />
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.<br />
** Cons:<br />
*** Interface more complicated and crowded<br />
*** More difficult for users to grasp?<br />
*** No immediate use case<br />
*** Makes no union type for tuple parameters a bigger problem.<br />
<br />
* [TE] Publish user survey?<br />
** No response from vistrails release yet<br />
** Not announced on vistrails-users yet.<br />
<br />
=== May 4, 2016 ===<br />
'''Updates'''<br />
* VisTrails 2.2.4 released<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
<br />
=== April 27, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Wrapping Bokeh<br />
** New method type "operation"<br />
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output<br />
**** Glyph sets function name and arguments as output<br />
***** Not serializable, but should always be computed together with the subject of the operation.<br />
**** Figure applies Glyph operation on itself<br />
*** Useful for matplotlib?<br />
**** matplotlib currently find actors by searching upstream?<br />
**** But this is more difficult since matplotlib is not function-based.<br />
** New "union" port type<br />
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value<br />
**** Uses only first value in tuple<br />
**** Bokeh have many properties that support (list, string, Float)<br />
*** Does not work with depth=1 types (e.g. Float of depth 1)<br />
**** Turn Depth 1 types with known type into lists?<br />
<br />
* [TE] New bugfix release<br />
** Finalize survey<br />
<br />
=== April 20, 2016 ===<br />
'''Updates'''<br />
* Merged upgrade fixes<br />
** Now has infinite loop test<br />
** loop test now correctly loops 5 times<br />
*** remap_module used wrong (current) package version when creating ModuleDescriptor<br />
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])<br />
<br />
'''Items to Discuss'''<br />
* New bugfix release<br />
** Merge usagestats<br />
** Need to finalize survey<br />
<br />
=== April 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?<br />
<br />
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])<br />
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]<br />
** Cherry-pick test to preserve source?<br />
** Need maxPipelineFixAttempts option?<br />
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.<br />
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?<br />
*** Would need to be moved into handle_invalid_pipeline<br />
<br />
=== April 6, 2016 ===<br />
'''Updates'''<br />
* [TE] Query View improvements<br />
** Done except for displaying "Version Info" and #1164<br />
* [RR] Working on finishing new interpreter (finally!)<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Try latest upgrade first, then previous.<br />
** If all fail, select latest upgrade?<br />
<br />
=== March 30, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Query View improvements<br />
** Unified do_version_switch and get_upgrades as validate_version()<br />
** Problems with query view using its own controller<br />
*** Fixed module info not being displayed for query results<br />
*** Still no version info for query results<br />
*** Still an issue with version query view not being updated<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== 2015 ==<br />
<br />
=== December 30, 2015 ===<br />
'''Updates'''<br />
* Bokeh package finished<br />
** Can output to Browser/IPython/Spreadsheet (Limited functionality in spreadsheet)<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Bokeh needs QWebEngine (Chromium) in Qt5 which requires PyQt5<br />
*** Many plotting libraries only work in browsers<br />
*** But Chromium may be a big and controversial dependency<br />
*** Or create a browser version of the spreadsheet?<br />
** Some users have requested Python 3<br />
** Not backwards compatible?<br />
** Target for VisTrails 3.0 with new interpreter?<br />
<br />
=== December 16, 2015 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Users list question<br />
* [TE] Wrapping Bokeh<br />
** Added bokeh property parser<br />
*** complements numpydoc parser<br />
*** Not all attributes are available, partially because I am using a pre-release version.<br />
** Wrapped modules<br />
*** bokeh.plotting finished (Backwards like matplotlib: Create plot then draw on it)<br />
*** bokeh.charts mostly done (Missing arguments)<br />
** QWebView mostly works (but no zooming or buttons)<br />
*** It would be better to use external browser or QWebEngineView<br />
** Added improvements to wrapper<br />
*** Generalized port translations (Color/Path to native types)<br />
*** Can use multiple docstring/property parsers simultaneously<br />
<br />
=== December 9, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Looking into wrapper<br />
** Will port TensorFlow<br />
** Doing a pass on the wrapper code & doc<br />
*** UX improvements<br />
** Factorize high-level docstring parsers into core.wrapper? (sphinx, numpy, google)<br />
** Problem with name vs module_name (fails sklearn tests because class_by_name() is broken)<br />
** Looking into compressing specs, lazy registration of modules<br />
<br />
=== December 2, 2015 ===<br />
'''Updates'''<br />
* [TE] Package wrapping<br />
** Wrapped most of numpy/scipy using __all__<br />
** Added class attribute and method access for classes (Can add to class or to separate inspector module)<br />
** TODO: Document wrapping procedure<br />
<br />
'''Items to Discuss'''<br />
<br />
=== November 25, 2015 ===<br />
'''Updates'''<br />
* [RR] New interpreter is coming along<br />
** Everything is a stream<br />
** Task system, eventually work-stealing parallelism<br />
** Streams no longer advance in lock-step<br />
*** Allows for constructions like filter, join, sorted-merge<br />
** depth>1 still to be tested, no current plan to allow this through the module interface<br />
*** Through looping though, can do streams of streams<br />
<br />
'''Items to Discuss'''<br />
* [TE] Package wrapping<br />
** Wrapped most numpy functions (no polynomials)<br />
** and half of scipy classes and functions<br />
** Added functions manually from documentation (no parseable lists?)<br />
** Focusing on data flow constructs (no property/method access for classes)<br />
*** Using spec diff (empty for now) and dynamic parser (stores spec in .vistrails/)<br />
** Python class/function wrapper now stable<br />
** Add google doc parser?<br />
*** Is there a parser available?<br />
<br />
=== November 18, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Package wrapping<br />
** Numpy class wrapping works<br />
*** Basic type is List, since most types are array_likes.<br />
*** TODO: More classes and functions<br />
*** Some operations are in-place and some aren't, and docstring not super clear<br />
*** [RR] It seems only methods mutate, and most have a numpy.xx function equivalent; just manually go through the methods and remove the mutating ones?<br />
** Class wrapper is modular<br />
*** Docstring parser and type string parser can be customized<br />
*** Classes can have optional attribute/method ports<br />
**** Inspectors and attribute/method modules can be created separately<br />
** Use PythonCalc as an example<br />
*** Can be re-implemented as a function with a parseable docstring<br />
*** [RR] PythonCalc is just an example, and it might actually make sense to get rid of it, or do a proper math package with scalar operations (as separate modules, no combobox)<br />
** [RR] TensorFlow to use wrapping as well<br />
*** Very simple (only types 'tensor' & 'variable')<br />
<br />
=== November 11, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Package wrapping<br />
** Wrapping numpy's classes using numpydoc (ndarray)<br />
*** Wrapping constructor arguments, attribute getters/setters as module ports<br />
**** Then how to access attributes afterwards?<br />
***** Use input value port to class modules?<br />
*** Wrap methods as modules? E.g., `ndarray.shape`.<br />
*** Function wrapping is almost a subset of class wrapping<br />
**** May be able to use the same parsing/execution methods<br />
** VTK's non-getter/setter methods could be wrapped as modules<br />
*** We could then remove extra logic in interpreter for keeping function order<br />
** Bokeh uses autogenerated docstrings<br />
*** We may be able to read the specification directly<br />
<br />
* [RR] [https://github.com/VisTrails/VisTrails/pull/1138 TensorFlow package]<br />
** Basic setup working, can execute the Mandelbrot example<br />
** Will autogenerate the operations<br />
<br />
=== November 4, 2015 ===<br />
'''Updates'''<br />
* [RR] Interpreter work requires a fix for [https://github.com/VisTrails/VisTrails/issues/1137 DB issue #1137]<br />
* [DK] Kitware's Resonant [http://www.kitware.com/source/home/post/173]<br />
** [https://github.com/girder/girder Girder]: data management system<br />
** [https://github.com/Kitware/romanesco Romanesco]: execution engine, uses [http://www.celeryproject.org/ Celery] for task management<br />
** Resonant Flow: web app for editing and executing workflows<br />
<br />
'''Items to Discuss'''<br />
* [TE] Added OSX Lion (10.7) VM on build machine using vagrant (seems ok with license?)<br />
** New builds works on Lion<br />
** Change the minimum requirement to 10.7?<br />
* [TE] Package wrapping<br />
* Added upgrade suggestions using 2 spec versions ([http://pastebin.com/30XUU8Gp example])<br />
** using name edit distance to find matches<br />
** Could use something better like port similarity for modules, and type similarity for ports<br />
** Should write python upgrade code?<br />
** Added spec to sklearn package<br />
*** Can now diff spec versions and keep the spec static<br />
*** There will be problems with downgrades and package versions<br />
<br />
=== October 29, 2015 ===<br />
'''Updates'''<br />
* [TE] nightly-build-mac fixed (Needed to approve xcode license)<br />
<br />
'''Items to Discuss'''<br />
* [TE] matplotlib wrapper<br />
** ported to general spec<br />
** Diff can be used on general spec<br />
*** Only needed minor changes<br />
*** Fixed indexing bug that corrupted specs<br />
*** Can now move diff tools to core/wrapper<br />
** Add new plots?<br />
*** New plots in mpl 1.3: eventplot<br />
*** New plots in mpl 1.4: angle_spectrum, magnitude_spectrum, phase_spectrum', violinplot<br />
*** No new plots in mpl 1.5 (from looking at boilerplate.py)<br />
<br />
=== October 21, 2015 ===<br />
'''Updates'''<br />
* [RR] Alexis has arrived, will be working with [RR] on a more efficient interpreter<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1133 New matplotlib package]<br />
** Supporting multiple package versions<br />
*** Check which version can be loaded [https://github.com/VisTrails/VisTrails/pull/1135 #1135]<br />
*** We cannot show version requirements in the list of packages because old packages are loaded when importing the codepath<br />
** Visual diff does not work well, but we can use the spec differ to see differences.<br />
** Unifying vtk and matplotlib wrapping specs (WIP)<br />
*** And porting matplotlibs spec differ to work on the general spec<br />
*** Spec differ can then be used on vtk and other packages in the future <br />
*** Will enable automatic upgrade generation<br />
* [RR] Rework interpreter<br />
** restore abstraction between interpreter/module code<br />
** build looping, streaming, caching into the interpreter<br />
** rework cache<br />
** look into parallelism once basic functionality is there<br />
<br />
=== October 14, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1133 New matplotlib package] (adding numpydoc parser)<br />
<br />
=== October 7, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://github.com/VisTrails/VisTrails/tree/fix-matplotlib-parser New matplotlib parser] (adding numpydoc parser)<br />
** Added numpydoc attribute parser for plots.<br />
*** TODO: Need better port spec reconciliation with call signature parser.<br />
*** We can create a general numpydoc parser, but attribute types are unique to matplotlib.<br />
** Needs package versioning<br />
*** New spec will not support matplotlib < v1.4 due to changed path to axes classes.<br />
*** Load spec version corresponding to installed matplotlib version?<br />
*** Need version downgrades?<br />
<br />
=== September 30, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Updating Matplotlib parser<br />
** Matplotlib docstring parser fails on numpy docstrings<br />
** I have [https://github.com/VisTrails/VisTrails/commit/eae8b0c16260810587e47c41cab53552cbb3cf0a added basic numpydoc parsing]<br />
*** Only used by a few docstrings so far<br />
*** Parsing uses many sources (class tables, signatures, ACCEPTS, method docstrings, definition parsing).<br />
** Caswell said they were thinking about moving to traitlets, but this is not ready yet<br />
<br />
=== September 23, 2015 ===<br />
'''Updates'''<br />
* [TE] VisTrails 2.2.3 released<br />
** Also have pushed to PyPI, binstar, etc.<br />
** Sourceforge vs. GitHub<br />
*** should be able to host releases on GitHub<br />
*** nightly binaries pushed to sf each night<br />
*** old binaries?<br />
* [TE] Re-wrapping MatPlotlib<br />
** Keep static generation - Docstrings are brittle beween matplotlib versions<br />
** Keep generating the executable classes - but create functions, not vistrails Modules <br />
*** Why is there so much patching?<br />
** Re-implemented Alternate PortSpec for InputPortSpec<br />
*** It will now inherit specs from parent (No need to reimplement)<br />
** Module upgrades?<br />
<br />
'''Items to Discuss'''<br />
<br />
=== September 16, 2015 ===<br />
'''Updates'''<br />
* [TE] Job Monitor tests and documentation done<br />
** OK to leave document package in packages directory?<br />
<br />
'''Items to Discuss'''<br />
* [TE] Library wrapping: How to do code patching? [https://docs.google.com/drawings/d/1Sh3eOXot6v2y880jHKQrqXqAwr-0aoqYligg3z8G8MY/edit?usp=sharing Wrapping diagram]<br />
** Insert code into module template (Old VTK code)<br />
** Patch the library that is called (New VTK code)<br />
** Somehow store code in spec and apply when executed?<br />
*** Executed code needs at a minimum access to inputs, outputs, current class<br />
*** Use standard input/output dict and obj reference that the code operates on?<br />
*** Can this be done while keeping the execution abstract?<br />
<br />
* Do new release now?<br />
<br />
=== September 9, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* New release?<br />
** [RR] wants reprounzip<br />
** tej changes & doc<br />
** job monitor & jobmixin fixes<br />
** mongodb<br />
** reprounzip<br />
** warning: MplFigure type is int<br />
** tabledata: add Reshape, DictoToTable & ListToTable, fix ListToTable with numpy arrays<br />
** PythonSource can have same name for an input & output<br />
** don't show spreadsheet at exit<br />
<br />
=== September 2, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Limit autosaves? [https://github.com/VisTrails/VisTrails/pull/1126 #1126]<br />
<br />
=== August 26, 2015 ===<br />
'''Updates'''<br />
* [TE] Added Job support to Parameter Explorations (Requested by Colin), and Mashups<br />
** specify job ids, need to specify different ids for parameter explorations since they have the same version id<br />
** how to deal with parameters passed in on command line<br />
* [General] provenance: should be creating a new version when we execute workflow with changed parameters?<br />
** currently, custom_params annotation stores this in provenance currently<br />
** [RR] API doesn't record provenance if passing in parameters: http://git.io/vsAA5<br />
'''Items to Discuss'''<br />
<br />
=== August 19, 2015 ===<br />
'''Updates'''<br />
* [RR] Improvements to tej, users' guide entry & example [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
* [RR] Internal docs [https://github.com/VisTrails/VisTrails/pull/1116 #1116]<br />
** On ReadTheDocs: [http://vistrails.readthedocs.org/ vistrails.readthedocs.org]<br />
* [TE] Working on Job Monitor documentation<br />
<br />
'''Items to Discuss'''<br />
* [RR] JobMixin and JobMonitor: stable now?<br />
** Definitely needs more tests<br />
* [TE] Stop testing VisTrails 2.0?<br />
<br />
=== August 12, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Internal docs [https://github.com/VisTrails/VisTrails/pull/1116 #1116]<br />
** On ReadTheDocs: [http://vistrails.readthedocs.org/ vistrails.readthedocs.org]<br />
* [RR] JobMixin and JobMonitor: stable now?<br />
** Definitely needs more tests<br />
* [RR] Improvements to tej, users' guide entry & example [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
* [RR] Build broken on Travis; because of IPython 4 released today? ([https://travis-ci.org/VisTrails/VisTrails/builds/75279795 build 992]; [https://github.com/VisTrails/VisTrails/issues/1123 #1123])<br />
<br />
=== August 5, 2015 ===<br />
'''Updates'''<br />
* [TE] Fixes to Jobs<br />
** Could not delete jobs<br />
** Could not run job in group<br />
** Job not reset when calling ModuleSuspended<br />
** Added deleting job from context menu<br />
* [RR] Writing documentation for everything [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
<br />
'''Items to Discuss'''<br />
<br />
=== July 29, 2015 ===<br />
'''Updates'''<br />
* RR still looking into new interpreter thing<br />
** Goal is to take out scheduling logic from Module so it can be split in multiple processes, and so that smarter strategies can be added in time<br />
** This means some work on packages<br />
** Spreadsheet can live in kernel process? Still some UI stuff that will take work (changing configuration, persistent archive's viewer, ...)<br />
* [TE] Problems running examples [https://github.com/VisTrails/VisTrails/pull/1111 #1111]<br />
** Testing of more examples requires additional packages on the test machines.<br />
** Fixed faulty line-ending in PythonSource:s failing on Python 2.6.<br />
** Test suite now testing SUDSWebServices (If web service is down, test suite will fail)<br />
** preferences.py test failed reloading 'dialogs' package, switched to using 'URL'.<br />
<br />
'''Items to Discuss'''<br />
<br />
=== July 22, 2015 ===<br />
'''Updates'''<br />
* [TE] mailing lists back online<br />
* [TE] Working on [https://github.com/VisTrails/VisTrails/issues/1107 #1107]<br />
* [RR] Working on ReproZip package <br />
<br />
'''Items to Discuss'''<br />
* [RR] MongoDB package [https://github.com/VisTrails/VisTrails/pull/1106 #1106]<br />
* [RR] Example for tej docs? [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
<br />
=== July 15, 2015 ===<br />
'''Updates'''<br />
* 2.2.2 released<br />
<br />
'''Items to Discuss'''<br />
* RR is considering executing everything in an IPython kernel (i.e. separate Python interpreter, like the one spawned for a notebook)<br />
** It's a separate process that we can restart without restarting the app/gui<br />
** We can isolate the execution environment (e.g. for the server)<br />
** We can run the whole thing remotely (if your data is elsewhere, just run VisTrails locally and the pipeline elsewhere)<br />
** Pipeline execution no longer makes the interface hang, it just makes the kernel hang (but that's fine)<br />
** We can use notebooks as modules (probably way nicer than the PythonSource module)<br />
** We can run multiple kernels so long as the ports carry things that are serializable<br />
*** meaning we can put the multithreaded-interpreter without all the hacky parts it has now<br />
*** we can run IPython kernels in all the languages IPython supports, [https://github.com/ipython/ipython/wiki/IPython-kernels-for-other-languages currently 46]<br />
<br />
=== July 8, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Buildbot github hook not working after IP address change<br />
* [TE] No reference to VistrailsApplication ([https://github.com/VisTrails/VisTrails/issues/1103 #1103])<br />
** [RR] Added PR [https://github.com/VisTrails/VisTrails/pull/1104 #1104]<br />
* [TE] Reopening VT file after saving with bundled subworkflow won't offer subworkflow upgrade ([https://github.com/VisTrails/VisTrails/issues/1102 #1102])<br />
** allow manual delete to fix right now<br />
** fix this on top of the use-uuid branch<br />
* [TE] Release VisTrails 2.2.1? ([https://github.com/VisTrails/VisTrails/blob/v2.2/CHANGELOG CHANGELOG])<br />
** [https://github.com/VisTrails/VisTrails/pull/1093 output-modules-use-global-dir]<br />
** [https://github.com/VisTrails/VisTrails/milestones/version%202.2 Current issues for 2.2]<br />
<br />
=== July 1, 2015 ===<br />
'''Updates'''<br />
* [TE] PROV fixed<br />
* [TE] Working on subworkflow issues<br />
* [RR] Considering reworking the controller (log vs exception problem, retained upgrades causing interferences) and also the interpreter (IPython?)<br />
<br />
'''Items to Discuss'''<br />
<br />
=== June 24, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* UV-CDAT<br />
* [TE] Can a cyclic workflow be valid? ([https://github.com/VisTrails/VisTrails/issues/1097 #1097])<br />
** focus on disabling the ability to create cyclic pipelines because more things break than just this with a cyclic pipeline<br />
* [RR] Relative paths ([https://github.com/VisTrails/VisTrails/pull/1057 #1057])<br />
** This interacts with the new bundle; how do handle packing files inside the VT bundle?<br />
<br />
=== June 17, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] current_version and reusing existing upgrades are broken ([https://github.com/VisTrails/VisTrails/issues/1095 ticket #1095])<br />
** Could be that export to PROV is using an unflushed upgrade pipeline?<br />
** current_version would then be correct.<br />
** It may work to flush the actions before exporting?<br />
<br />
=== June 10, 2015 ===<br />
'''Updates'''<br />
* DAT: fixed VTK issue on Linux and Mac<br />
* Still crashes on Windows. Need help! Reminder: this works in the VisTrails spreadsheet (QCellPresenter), although no widget get changed there during a drag<br />
** Is it a VTK bug?<br />
** Is it simply impossible to change widgets during the drag, should we do it a different way?<br />
** Did I miss something that is done in VisTrails but somehow not in DAT?<br />
** Low prio, UV-CDAT doesn't run on Windows anyway<br />
<br />
'''Items to Discuss'''<br />
* [TE] Executable Paper ([https://github.com/VisTrails/VisTrails/issues/1086 ticket #1088] [https://github.com/VisTrails/VisTrails/pull/1088 Updated pull request])<br />
* Single instance code and batch mode<br />
<br />
=== June 3, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Executable Paper ([https://github.com/VisTrails/VisTrails/issues/1086 ticket #1088] [https://github.com/VisTrails/VisTrails/pull/1088 pr])<br />
** Requires fixes to command line parameters, Output modules, and batch mode<br />
** How to test this<br />
** Updated missing/outdated flags<br />
** Fixed view issues when generating graphs<br />
* [TE] batch mode<br />
** SpreadsheetOutput not enabled in batch mode. Should we check is_running_gui instead?<br />
** Other instance setting flags from caller<br />
** Is graphsAsPdf replacing spreadsheetDumpPdf?<br />
** graphsAsPdf true by Default?<br />
** Batch mode executing by default (Not needed when generating graphs)<br />
** Re-added workflowInfo as withWorkflowInfo for writing graph and xml workflow<br />
** [DK] batch mode should be outputting to files or stdout, shouldn't always trigger SpreadsheetMode<br />
** execute flag, maybe make execution the default and allow a "--no-execute" if you only want to capture graphs, for example<br />
<br />
=== May 27, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] API question<br />
** http://lists.vistrails.org/pipermail/vistrails-users/2015-May/000357.html<br />
* Status of DAT<br />
** Can add VCS plot to DAT but need to work on configuration windows<br />
** Documentation for graphics templates for UV-CDAT/vcs?<br />
* Qt support?<br />
<br />
=== May 20, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Reviving the DAT, integrating scripting & porting UV-CDAT!<br />
** On [https://github.com/VisTrails/DAT GitHub] ([https://github.com/VisTrails/DAT/issues issues])<br />
** Merging 2 years of development taking longer than expected, but getting there. The plan is to get the patches in VisTrails and never fork again, we never want to get in UV-CDAT's situation (and don't need to).<br />
** VTK cell works fine on Linux but there was flickering on Mac & Windows before; still issues on Mac (Windows status unknown)<br />
** Can get a VCS plot soon (but will need VTK cell fix)<br />
** Integrate in UV-CDAT's build system (so we have cdms, VCS, ...) -> RR can do this, low priority<br />
** How do we integrate scripting?<br />
*** We want to be able to seamlessly make changes to a plot by changing Python code<br />
*** Define new plots by entering Python code without writing modules/packages?<br />
* [RR] UV-CDAT [https://github.com/UV-CDAT/uvcdat/issues/1275 needs] [https://github.com/VisTrails/VisTrails/pull/1073 #1073], please take a look<br />
* [RR] Subworkflow issues<br />
** [https://github.com/VisTrails/VisTrails/issues/1065 #1065]: missing attribute (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/1066 #1066]: groups in subworkflows were ignored when finding dependencies (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/725 #725]: "upgrade subworkflow" button not popping up (fixed); also, namespace issue<br />
** [https://github.com/VisTrails/VisTrails/issues/1071 #1071]: subworkflows don't go through upgrades<br />
* [RR] [https://github.com/VisTrails/VisTrails/issues/1074 #1074]: can't load and edit a single pipeline if upgrade happen<br />
<br />
=== May 13, 2015 ===<br />
'''Updates'''<br />
* [RR] Test skipping whitelist ([https://github.com/VisTrails/VisTrails/pull/1069 #1069]) -- low priority<br />
* [RR] Custom matplotlib modules can't be compatible with both 2.1 and 2.2 ([https://github.com/VisTrails/VisTrails/issues/1067 #1067]); should be fixed for ALPS ([https://github.com/VisTrails/VisTrails/issues/1070 #1070])<br />
<br />
'''Items to Discuss'''<br />
* [RR] UV-CDAT [https://github.com/UV-CDAT/uvcdat/issues/1275 needs] [https://github.com/VisTrails/VisTrails/pull/1073 #1073], please take a look<br />
* [RR] Subworkflow issues<br />
** [https://github.com/VisTrails/VisTrails/issues/1065 #1065]: missing attribute (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/1066 #1066]: groups in subworkflows were ignored when finding dependencies (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/725 #725]: "upgrade subworkflow" button not popping up (fixed); also, namespace issue<br />
** [https://github.com/VisTrails/VisTrails/issues/1071 #1071]: subworkflows don't go through upgrades<br />
* [RR] [https://github.com/VisTrails/VisTrails/issues/1074 #1074]: can't load and edit a single pipeline if upgrade happen<br />
<br />
=== May 7, 2015 ===<br />
'''Updates'''<br />
* [RR] Export/import workflow to Python working!<br />
<br />
'''Items to Discuss'''<br />
* [Claudio] UV-CDAT<br />
** The UV-CDAT project is the biggest user base of VisTrails<br />
** VisTrails package management provides a lot of friction towards people plugging in their code<br />
** Need to make it easy to integrate your random Python scripts in the system without having to deal with all the boilerplate, at least in the first step<br />
** [RR] argues that modules are still good; UV-CDAT shouldn't move towards a purely script-based backend<br />
** [RR] export/import with Python could reduce a lot of that friction by allowing 1) to edit workflow as Python 2) to open up boxes automatically if needed code doesn't match actual modules<br />
** ...<br />
<br />
=== April 29, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] <strike>Abstractions</strike> subworkflows status ([https://github.com/VisTrails/VisTrails/labels/subworkflows tickets])<br />
* [RR] matplotlib compatibility (2.1 & 2.2), [https://github.com/VisTrails/VisTrails/issues/1067 #1067]<br />
** RR to try and fix ALPS matplotlib modules<br />
<br />
=== April 22, 2015 ===<br />
'''Updates'''<br />
* 2.2.0 has been released!<br />
* Windows issue (via email from Ryan)<br />
** issue with manifest file (may be a new file in VTK6?)<br />
** Tommy has regenerated new Windows builds<br />
* Binaries, pypi, and conda released<br />
* [RR] Export as script<br />
* Python sources using VTK need to switch to SetInputData (users should be aware of this)<br />
<br />
=== April 15, 2015 ===<br />
'''Updates'''<br />
* Ready for 2.2.0 (apart from binary/deps issues)<br />
** Missing some libs (scikit-learn, tej, tdparser, SQLAlchemy+connectors<br />
** Windows: runvistrails.py is no longer used, so the PATH is wrong<br />
** Windows: pip is broken, but it probably wouldn't work anyway because of permission issues (disable this?)<br />
* Queries, upgrades and getPipeline() usage ([https://github.com/VisTrails/VisTrails/issues/1054 #1054])<br />
** Getting a pipeline with getPipeline() is not safe: it might return an invalid pipeline<br />
** This is used in many places throughout the code, like queries<br />
** Upgrading would require going through the controller, but that creates new actions<br />
* [TC] Avoid copying a module's output if it's used as input by exactly one downstream module ([https://github.com/VisTrails/VisTrails/issues/1060 #1060]) (useful for big numpy arrays you can update in-place)<br />
<br />
=== April 8, 2015 ===<br />
'''Updates'''<br />
* [DK] Merged RR's changes for output modules (1012 and 1013)<br />
** RR will merge remaining changes, then create v2.2 branch!<br />
<br />
'''Items to Discuss'''<br />
* BNL need numpy array to VTK image<br />
** Looks like VTK has some helpers for this<br />
** We will help if issues arise<br />
** Will contribute back to VisTrails package<br />
* Upgrade issue: [https://github.com/VisTrails/VisTrails/issues/1017 #1017]<br />
** Automatic upgrades should happen between versions of provided upgrades<br />
** Our existing upgrades work around this so it doesn't need to be 2.2.0<br />
* Corner-case VTK modules<br />
** No longer need VTKCell input port, so don't interfere with registry and API anymore<br />
** Still work weirdly, people probably shouldn't use them<br />
** But we have lots of clunky modules since we wrap the whole of VTK; some people might rely on this and know how to use them, let's keep them anyway<br />
** Ready for 2.2.0<br />
<br />
=== April 1, 2015 ===<br />
'''Updates'''<br />
* [RR] UV-CDAT: bugfixing for 2.2, long-term plans: implement scripting import/export in VisTrails, port to UV-CDAT<br />
* Possibly, try to move to regular VisTrails to use new features<br />
<br />
'''Items to Discuss'''<br />
* 2.2 release: nothing much is pending anymore, release next week?<br />
** [TE] vtkExporter classes [https://github.com/VisTrails/VisTrails/issues/1032 #1032]<br />
** [RR] Hiding upgrades in version tree might make it (or might be 2.2.1 so we can test it out) [https://github.com/VisTrails/VisTrails/issues/949 #949]<br />
** Output modules changes to go in<br />
* add note to documentation about order of parameters in VTK<br />
* add issue about exporter upgrades if not already there<br />
<br />
=== March 25, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* 2.2 release<br />
** Checklist on Github: https://github.com/VisTrails/VisTrails/wiki/2.2-Release-Checklist<br />
** Review output modules<br />
** RR has a couple more issues to fix<br />
** Ready to go -- sign app package for OSX? [https://github.com/VisTrails/VisTrails/issues/984 #984] We need access to the Apple Membership team<br />
<br />
=== March 18, 2015 ===<br />
'''Updates'''<br />
* [TE] VTK6 works<br />
<br />
'''Items to Discuss'''<br />
* [RR] Release v2.1.5 with backported tabledata?<br />
** MTA example needs updated tabledata (for JoinTables)<br />
** Google Maps package still not available<br />
* [RR] Work torward v2.2.0?<br />
** changes:<br />
*** new persistence<br />
*** API changes<br />
*** output module changes (upgrades?), maintain cells but try to upgrade<br />
*** not wrapping stuff<br />
*** VTK6? yes<br />
*** JobSubmission stuff?<br />
*** relabeling for upgrades [https://github.com/VisTrails/VisTrails/issues/949 #949]<br />
** makes sense, needs the tree view code to be updated, check selection<br />
** See [https://github.com/VisTrails/VisTrails/wiki/2.2-Release-Checklist 2.2 checklist]<br />
* Discussion of [https://github.com/VisTrails/VisTrails/issues/1016 #1016]<br />
** plumbing between outputs and output modes, how to define a mode that works for many outputs without writing for each output?<br />
<br />
=== March 11, 2015 ===<br />
'''Updates'''<br />
* [RR] persistent_archive done; merge? ([https://github.com/VisTrails/VisTrails/pull/755 #755])<br />
** note about the focus events for widgets<br />
** TE be aware of file_archive for future binaries that include persistent_archive<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/998 New VTK package] finished<br />
<br />
'''Items to Discuss'''<br />
* [RR] Unmark UV-CDAT/VisTrails as a fork of VisTrails/VisTrails? (see last week; decision needed)<br />
** RR email to JF about this<br />
** yes; email sent to Github<br />
* [RR] Switching order of output ports ([https://github.com/VisTrails/VisTrails/pull/1006 #1006])<br />
** added port specs are sorted at a separate spot (Module.*_port_specs properties) than those in the registry (which are sorted in the registry), but those two lists are just combined without respect to sort keys<br />
** need to determine whether the two lists should be merged or remain distinct<br />
** should make sure that order of input ports and output ports makes parallel connections for things with same order<br />
** DK suggests breaking backward compatibility here: workflows still run, can fix easily if a problem in an existing package.<br />
* [RR] Question about output modes ([https://github.com/VisTrails/VisTrails/pull/1007 #1007]), how to integrate in API ([https://github.com/VisTrails/VisTrails/pull/24 #24])<br />
** Should ImageFileMode be removed? ("image" is not a mode, "file" is)<br />
** ImageOutput missing?<br />
** Feel free to change how formats works<br />
* [TE] Test suite segfaults on Fedora 17 virtualbox. Install newer version? (Support ended in 2013)<br />
<br />
=== March 4, 2015 ===<br />
'''Updates'''<br />
* [TE] VTK wrapper<br />
** Works on VTK 5.10<br />
** Still need to test VTK 6<br />
** New general wrappers for python functions and classes <br />
<br />
'''Items to Discuss'''<br />
* [RR] Unmark UV-CDAT/VisTrails as a fork of VisTrails/VisTrails?<br />
** When [http://i.imgur.com/UTY2qfg.png filing a pull request] from UV-CDAT/VisTrails, VisTrails/VisTrails is selected by default<br />
** People keep forgetting to change the default ([https://github.com/VisTrails/VisTrails/pull/956 #956] [https://github.com/VisTrails/VisTrails/pull/968 #968] [https://github.com/VisTrails/VisTrails/pull/999 #999] [https://github.com/VisTrails/VisTrails/pull/1000 #1000] [https://github.com/VisTrails/VisTrails/pull/1003 #1003] [https://github.com/VisTrails/VisTrails/pull/1004 #1004] [https://github.com/VisTrails/VisTrails/pull/1005 #1005])<br />
** Only way to change that is to not have it marked as a fork of VisTrails/VisTrails<br />
** Github staff [https://gist.github.com/remram44/b364c24ef1e462ce6e2f can make that change for us]; should we do it?<br />
** Juliana: comments on this? visibility vs. convenience/annoyance for developers<br />
* [RR] What about [https://github.com/VisTrails/VisTrails/issues/991 #991]?<br />
<br />
=== February 25, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] vtkviewcell for infovis support, can we unify with VTKCell?<br />
** need to test this<br />
* [TE] vtk wrapping<br />
** Mostly finished<br />
** VTK 5.10 produce incorrect results with old wrapping<br />
*** Old wrapper is based mostly on VTK 4<br />
*** Most vtk_examples affected<br />
*** [http://www.vtk.org/Wiki/VTK/VTK_6_Migration/Replacement_of_SetInput]<br />
*** should be able to upgrade from SetInput to SetInputData (need to drop GetOutput and replace with self ports)<br />
*** can we change vtkInstance to just return self and not wrap things<br />
** terminator example not working under 5.8?<br />
* How does VTK wrapping fit into general wrapping framework?<br />
* [RR] new persistence package<br />
<br />
=== February 18, 2015 ===<br />
'''Updates'''<br />
* [RR] New VisTrails API and IPython integration ([https://github.com/VisTrails/VisTrails/pull/24 #24])<br />
<br />
'''Items to Discuss'''<br />
* [TE] VTK wrapping<br />
** [https://gist.github.com/rexissimus/16b74693545b5e7c1035 Benchmarking vtk package]<br />
*** Old: 24.7 seconds <br />
*** New: 10.5 seconds (Except first time that adds 8 sec)<br />
**** The parsing that calls is_abstract (that tries to instanciate all vtk classes) is now only run the first time.<br />
**** get_items_from_sigstring takes 2 seconds, maybe we can use a lookup dict for already computed sigstrings?<br />
** Now using a general python function wrapper<br />
*** VTK classes are wrapped into python function that does not depend on vistrails<br />
*** VTK functions can be executed without vistrails<br />
*** The spec maps functions into vistrails modules, but can also describe wrapping <br />
*** A general python function wrapper that supports <br />
**** kwarg inputs<br />
**** single, list, dict outputs<br />
**** callback for progress reporting<br />
**** temporary file generator for using FilePool<br />
**** optional output generation<br />
*** Creating specs:<br />
***** Create spec by hand<br />
***** Auto-create spec outline (TODO) and manually finish it<br />
***** Dynamically create spec (VTK)<br />
***** Implement documentation wrappers (Can use scikitlearn wrapper to wrap numpydoc) (TODO)<br />
***** Classes as bad functions needs to be wrapped in new functions before they are wrapped. This is different for each package.<br />
****** Classes is hard: Like VTK, and matplotlib. Scikit-learn does still not wrap classes<br />
***** Spec diffing and patching could be done using code from matplotlib.<br />
** Still needs upgrades from old VTK package<br />
*** Is it possible to dynamically wrap functions, e.g, you see a SetFunc and just remove the 'Set' prefix. Or do you need to create a complete mapping?<br />
<br />
=== February 11, 2015 ===<br />
'''Updates'''<br />
* Update from Friday's meeting<br />
** discussed VisTrails internals<br />
** discussed wrapping<br />
*** xml discussion, hard to modify because tied to db code<br />
*** TE has made it possible to add the schema-defined attributes to the intermediate representation<br />
*** higher-level operations on the port specs<br />
** make sure the simple case works<br />
*** [JF] take a simple package with documentation and figure out what the base case for wrapping is<br />
<br />
'''Items to Discuss'''<br />
* [TE] VTK wrapping<br />
** Dynamic loading works<br />
** Reading XML is fast enough, but [https://gist.github.com/rexissimus/8272d3c776ccfa115d82 unserializing data is slow]<br />
** Working on patterns for patching<br />
** Matplotlib has many advanced patterns like argument ordering, nested arguments, alternateSpecs, output types.<br />
*** Having all this in a general wrapper might confuse users?<br />
** [RR] Delay module (except for identifiers) until you need it---e.g. don't deal with port specs, etc. until necessary<br />
* Scripting Support [https://github.com/VisTrails/VisTrails/pull/950 #950]<br />
** [RR] Issue with getting code from modules<br />
** Design a simple solution<br />
** [JF] Couldn't you use modules as black boxes without conversion, just to call into modules/subworkflows easily from e.g. IPython?<br />
*** [RR] This is a job for the API, and a very separate use case. See [https://github.com/VisTrails/VisTrails/pull/24 #24]<br />
<br />
=== February 4, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* Wrapping<br />
** Format to use? Currently XML (like current matplotlib)<br />
*** JSON and YAML have simple "to python dictionary" methods<br />
*** But don't stream<br />
*** YAML a lot easier for humans<br />
** [DK] vtk-new-package also changes parameter names, creates enumerations<br />
*** intermediate schema needs to be extensible<br />
*** packages will want to store there specific infos for compute() method generation<br />
*** also might have specs-altering info, like matplotlib's alternateSpec<br />
** representation to code , registry already has schema for some aspects<br />
** [RR] We might want to see if Module subclasses can be created lazily<br />
*** no need to create all the classes just to register them in the registry and never actually use most of them<br />
*** future effort<br />
* [RR] Where should VisTrails packages live?<br />
** tej installs as 'vistrailspkg.tej', TE installed it as 'userpackages.tej'<br />
** Currently, standard packages are 'vistrails.packages.', user packages are 'userpackages.' and packages loaded through pkg_resources might be anything<br />
** [RR] Use 'vistrailspkg.' everywhere?<br />
** Long-term effort to simplify package distribution/installation (and have VisTrails get them automatically?)<br />
<br />
=== January 28, 2015 ===<br />
'''Updates'''<br />
* T. Caswell to come visit on Fri 6 to discuss wrapping work<br />
<br />
'''Items to Discuss'''<br />
* [TE] New VTK wrapping<br />
** Current code by DK seems a good deal faster<br />
** Generates XML that can be patched/tweaked, generates Python code from it, VisTrails only loads generated Python code<br />
** RR would rather have VisTrails load intermediate representation (= XML) directly, wants to make sure this is not slower<br />
** The goal is to turn the intermediate step into something generic that would be used for every wrapped package (vtk, numpy, matplotlib, sklearn, java) instead of each having its own<br />
** TC has its own code at [https://github.com/Nikea/VTTools/blob/master/vttools/wrap_lib.py github:VTTools] which parses numpy docstrings and generates modules, doesn't yet handle classes or persist anything<br />
* Web crawler<br />
** Right now, TE starts jobs for "start crawler", "stop crawler", "install classifier"<br />
** RR would rather have the crawling be a job as far as VisTrails and tej are concerned<br />
** The whole thing would be one pipeline: load examples, train classifier, start crawler [check for job, kill previous one, upload model, start processes], get snapshot, visualize<br />
*** Need some support in tej and job submission system: long-running jobs, stop a job (wait for it to finish?), restart a job even though results are cached<br />
<br />
=== January 21, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Unified wrapping method discussion [https://github.com/VisTrails/VisTrails/issues/991 #991]<br />
** TE to work on reusable method with intermediate representation, starting with VTK<br />
* [RR] Examples for scikit-learn: JF has an old example using Weka with parameter exploration (not currently in source tree)<br />
** AM's examples are enough<br />
* [AM] scikit-learn package is done, merge it in? [https://github.com/VisTrails/VisTrails/pull/955#issuecomment-70286788 #955]<br />
** RR will merge<br />
* [RR] What should copyright headers say? [https://github.com/VisTrails/VisTrails/pull/994#issuecomment-70173935 #994]<br />
** Let's keep everything in there: Utah/Poly/NYU<br />
<br />
=== January 14, 2015 ===<br />
'''Updates'''<br />
* [TE] Working on classifier<br />
* [RR] Scripting integration, work in progress<br />
<br />
'''Items to Discuss'''<br />
* [RR] Unified wrapping method discussion ([https://github.com/VisTrails/VisTrails/issues/991 #991])<br />
** Let's talk next week, [AM] and [DK] are not here<br />
<br />
=== January 7, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* make sure that we address critical issues, questions, and pending review branches in a timely manner<br />
* scripting support<br />
** [RR] no issues if we want to just keep annotations in the generated code to allow the link back to a workflow<br />
** [RR] can translate from workflow to script, working on script to workflow<br />
** will work for parameter value changes, structural changes require changes to the annotations<br />
** need to publish best practices here<br />
** would be cool to do looping in scripts (easier interface than with workflows)<br />
* notebook support (convert form notebook to workflow)<br />
** RR will sync with FC on this<br />
* Issue with console in built-from-scratch<br />
** [TC] iPython rearranged some of the completion stuff in 2.2 and 2.3<br />
** binary has old version of iPython -> 1.0.0, should we update?<br />
* [TC] automated wrapping of numpy and scipy<br />
** discovered a bunch of malformed documentation in numpy and scipy<br />
** has github repo for vistrails tools<br />
** example modules wrap a bunch of R stuff (not baked in, just how things are)<br />
** will be pushing wrapping logic up<br />
** port names forbidden (window and domain)<br />
** have an import hook to get from yaml directly to VisTrails Modules<br />
** should work for any python modules with well-formed numpy docstrings.<br />
* [Action] should make it clear in documentation that Constant now means serializable not that the value doesn't change (e.g. List)<br />
* [TC] might be interesting to try to build components of matplotlib and accumulate in figure (long-term project, but thinking about how this might work)<br />
* [TE] build and build scripts<br />
** completely automatic, buildbot<br />
** need to set the build machines for the environment we want for the binary<br />
** would virtualenv work here?<br />
** [TC] anaconda can pin versions, potential path to test different configurations<br />
** Q: upload nightly binary builds? A: makes sense, make sure they are well-labeled<br />
* sourceforge stats: e.g. http://sourceforge.net/projects/vistrails/files/vistrails/nightly/vistrails-src-nightly.tar.gz/stats/timeline?dates=2014-01-01+to+2015-01-07<br />
* package issues (see Remi's message)<br />
* [TE] Scope of tej<br />
** Support single ssh commands?<br />
** Queue can be used as a remote machine (crawler is using queue.call*)<br />
* SourceForge stats: http://sourceforge.net/projects/vistrails/files/vistrails/nightly/vistrails-src-nightly.tar.gz/stats/timeline?dates=2014-01-01+to+2015-01-07<br />
<br />
== Older meetings ==<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=11859Development2016-05-25T14:35:03Z<p>Remi: /* May 25, 2016 */</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2016 ==<br />
<br />
=== May 25, 2016 ===<br />
<br />
'''Updates'''<br />
* Mailing-list fixed<br />
* Getting stats from the "server news" mechanism<br />
<br />
'''Items to Discuss'''<br />
<br />
=== May 18, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] mailing list still down - Julio working on it<br />
* [TE] Improved union port<br />
** New schema 1.0.5 adds portitem.union string attribute<br />
** Works with pipeline view, parameter exploration view, and mashup view<br />
** Should work well for matplotlib<br />
** Use with VTK ports?<br />
*** It uses [item_1,item_2] style ports usually with different tuple length.<br />
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]<br />
<br />
=== May 11, 2016 ===<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?<br />
*** Shows single port for all ports with same sort_key<br />
*** Select specific type when adding/showing<br />
*** Only one type should be visible on the module at any time<br />
*** Making tuple parameters union types is probably too complicated<br />
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly) <br />
<br />
* [TE] Feature suggestion: Let connections connect to functions/parameters<br />
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters<br />
** Use separate views for ports and functions (like before)<br />
*** Allow rearranging of functions<br />
** Pros:<br />
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.<br />
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.<br />
**** Makes List module simpler<br />
*** Connect to parameter in tuple<br />
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds. <br />
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.<br />
** Cons:<br />
*** Interface more complicated and crowded<br />
*** More difficult for users to grasp?<br />
*** No immediate use case<br />
*** Makes no union type for tuple parameters a bigger problem.<br />
<br />
* [TE] Publish user survey?<br />
** No response from vistrails release yet<br />
** Not announced on vistrails-users yet.<br />
<br />
=== May 4, 2016 ===<br />
'''Updates'''<br />
* VisTrails 2.2.4 released<br />
<br />
'''Items to Discuss'''<br />
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]<br />
<br />
=== April 27, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Wrapping Bokeh<br />
** New method type "operation"<br />
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output<br />
**** Glyph sets function name and arguments as output<br />
***** Not serializable, but should always be computed together with the subject of the operation.<br />
**** Figure applies Glyph operation on itself<br />
*** Useful for matplotlib?<br />
**** matplotlib currently find actors by searching upstream?<br />
**** But this is more difficult since matplotlib is not function-based.<br />
** New "union" port type<br />
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value<br />
**** Uses only first value in tuple<br />
**** Bokeh have many properties that support (list, string, Float)<br />
*** Does not work with depth=1 types (e.g. Float of depth 1)<br />
**** Turn Depth 1 types with known type into lists?<br />
<br />
* [TE] New bugfix release<br />
** Finalize survey<br />
<br />
=== April 20, 2016 ===<br />
'''Updates'''<br />
* Merged upgrade fixes<br />
** Now has infinite loop test<br />
** loop test now correctly loops 5 times<br />
*** remap_module used wrong (current) package version when creating ModuleDescriptor<br />
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])<br />
<br />
'''Items to Discuss'''<br />
* New bugfix release<br />
** Merge usagestats<br />
** Need to finalize survey<br />
<br />
=== April 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?<br />
<br />
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])<br />
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]<br />
** Cherry-pick test to preserve source?<br />
** Need maxPipelineFixAttempts option?<br />
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.<br />
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?<br />
*** Would need to be moved into handle_invalid_pipeline<br />
<br />
=== April 6, 2016 ===<br />
'''Updates'''<br />
* [TE] Query View improvements<br />
** Done except for displaying "Version Info" and #1164<br />
* [RR] Working on finishing new interpreter (finally!)<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Try latest upgrade first, then previous.<br />
** If all fail, select latest upgrade?<br />
<br />
=== March 30, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Query View improvements<br />
** Unified do_version_switch and get_upgrades as validate_version()<br />
** Problems with query view using its own controller<br />
*** Fixed module info not being displayed for query results<br />
*** Still no version info for query results<br />
*** Still an issue with version query view not being updated<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== 2015 ==<br />
<br />
=== December 30, 2015 ===<br />
'''Updates'''<br />
* Bokeh package finished<br />
** Can output to Browser/IPython/Spreadsheet (Limited functionality in spreadsheet)<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Bokeh needs QWebEngine (Chromium) in Qt5 which requires PyQt5<br />
*** Many plotting libraries only work in browsers<br />
*** But Chromium may be a big and controversial dependency<br />
*** Or create a browser version of the spreadsheet?<br />
** Some users have requested Python 3<br />
** Not backwards compatible?<br />
** Target for VisTrails 3.0 with new interpreter?<br />
<br />
=== December 16, 2015 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Users list question<br />
* [TE] Wrapping Bokeh<br />
** Added bokeh property parser<br />
*** complements numpydoc parser<br />
*** Not all attributes are available, partially because I am using a pre-release version.<br />
** Wrapped modules<br />
*** bokeh.plotting finished (Backwards like matplotlib: Create plot then draw on it)<br />
*** bokeh.charts mostly done (Missing arguments)<br />
** QWebView mostly works (but no zooming or buttons)<br />
*** It would be better to use external browser or QWebEngineView<br />
** Added improvements to wrapper<br />
*** Generalized port translations (Color/Path to native types)<br />
*** Can use multiple docstring/property parsers simultaneously<br />
<br />
=== December 9, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Looking into wrapper<br />
** Will port TensorFlow<br />
** Doing a pass on the wrapper code & doc<br />
*** UX improvements<br />
** Factorize high-level docstring parsers into core.wrapper? (sphinx, numpy, google)<br />
** Problem with name vs module_name (fails sklearn tests because class_by_name() is broken)<br />
** Looking into compressing specs, lazy registration of modules<br />
<br />
=== December 2, 2015 ===<br />
'''Updates'''<br />
* [TE] Package wrapping<br />
** Wrapped most of numpy/scipy using __all__<br />
** Added class attribute and method access for classes (Can add to class or to separate inspector module)<br />
** TODO: Document wrapping procedure<br />
<br />
'''Items to Discuss'''<br />
<br />
=== November 25, 2015 ===<br />
'''Updates'''<br />
* [RR] New interpreter is coming along<br />
** Everything is a stream<br />
** Task system, eventually work-stealing parallelism<br />
** Streams no longer advance in lock-step<br />
*** Allows for constructions like filter, join, sorted-merge<br />
** depth>1 still to be tested, no current plan to allow this through the module interface<br />
*** Through looping though, can do streams of streams<br />
<br />
'''Items to Discuss'''<br />
* [TE] Package wrapping<br />
** Wrapped most numpy functions (no polynomials)<br />
** and half of scipy classes and functions<br />
** Added functions manually from documentation (no parseable lists?)<br />
** Focusing on data flow constructs (no property/method access for classes)<br />
*** Using spec diff (empty for now) and dynamic parser (stores spec in .vistrails/)<br />
** Python class/function wrapper now stable<br />
** Add google doc parser?<br />
*** Is there a parser available?<br />
<br />
=== November 18, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Package wrapping<br />
** Numpy class wrapping works<br />
*** Basic type is List, since most types are array_likes.<br />
*** TODO: More classes and functions<br />
*** Some operations are in-place and some aren't, and docstring not super clear<br />
*** [RR] It seems only methods mutate, and most have a numpy.xx function equivalent; just manually go through the methods and remove the mutating ones?<br />
** Class wrapper is modular<br />
*** Docstring parser and type string parser can be customized<br />
*** Classes can have optional attribute/method ports<br />
**** Inspectors and attribute/method modules can be created separately<br />
** Use PythonCalc as an example<br />
*** Can be re-implemented as a function with a parseable docstring<br />
*** [RR] PythonCalc is just an example, and it might actually make sense to get rid of it, or do a proper math package with scalar operations (as separate modules, no combobox)<br />
** [RR] TensorFlow to use wrapping as well<br />
*** Very simple (only types 'tensor' & 'variable')<br />
<br />
=== November 11, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Package wrapping<br />
** Wrapping numpy's classes using numpydoc (ndarray)<br />
*** Wrapping constructor arguments, attribute getters/setters as module ports<br />
**** Then how to access attributes afterwards?<br />
***** Use input value port to class modules?<br />
*** Wrap methods as modules? E.g., `ndarray.shape`.<br />
*** Function wrapping is almost a subset of class wrapping<br />
**** May be able to use the same parsing/execution methods<br />
** VTK's non-getter/setter methods could be wrapped as modules<br />
*** We could then remove extra logic in interpreter for keeping function order<br />
** Bokeh uses autogenerated docstrings<br />
*** We may be able to read the specification directly<br />
<br />
* [RR] [https://github.com/VisTrails/VisTrails/pull/1138 TensorFlow package]<br />
** Basic setup working, can execute the Mandelbrot example<br />
** Will autogenerate the operations<br />
<br />
=== November 4, 2015 ===<br />
'''Updates'''<br />
* [RR] Interpreter work requires a fix for [https://github.com/VisTrails/VisTrails/issues/1137 DB issue #1137]<br />
* [DK] Kitware's Resonant [http://www.kitware.com/source/home/post/173]<br />
** [https://github.com/girder/girder Girder]: data management system<br />
** [https://github.com/Kitware/romanesco Romanesco]: execution engine, uses [http://www.celeryproject.org/ Celery] for task management<br />
** Resonant Flow: web app for editing and executing workflows<br />
<br />
'''Items to Discuss'''<br />
* [TE] Added OSX Lion (10.7) VM on build machine using vagrant (seems ok with license?)<br />
** New builds works on Lion<br />
** Change the minimum requirement to 10.7?<br />
* [TE] Package wrapping<br />
* Added upgrade suggestions using 2 spec versions ([http://pastebin.com/30XUU8Gp example])<br />
** using name edit distance to find matches<br />
** Could use something better like port similarity for modules, and type similarity for ports<br />
** Should write python upgrade code?<br />
** Added spec to sklearn package<br />
*** Can now diff spec versions and keep the spec static<br />
*** There will be problems with downgrades and package versions<br />
<br />
=== October 29, 2015 ===<br />
'''Updates'''<br />
* [TE] nightly-build-mac fixed (Needed to approve xcode license)<br />
<br />
'''Items to Discuss'''<br />
* [TE] matplotlib wrapper<br />
** ported to general spec<br />
** Diff can be used on general spec<br />
*** Only needed minor changes<br />
*** Fixed indexing bug that corrupted specs<br />
*** Can now move diff tools to core/wrapper<br />
** Add new plots?<br />
*** New plots in mpl 1.3: eventplot<br />
*** New plots in mpl 1.4: angle_spectrum, magnitude_spectrum, phase_spectrum', violinplot<br />
*** No new plots in mpl 1.5 (from looking at boilerplate.py)<br />
<br />
=== October 21, 2015 ===<br />
'''Updates'''<br />
* [RR] Alexis has arrived, will be working with [RR] on a more efficient interpreter<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1133 New matplotlib package]<br />
** Supporting multiple package versions<br />
*** Check which version can be loaded [https://github.com/VisTrails/VisTrails/pull/1135 #1135]<br />
*** We cannot show version requirements in the list of packages because old packages are loaded when importing the codepath<br />
** Visual diff does not work well, but we can use the spec differ to see differences.<br />
** Unifying vtk and matplotlib wrapping specs (WIP)<br />
*** And porting matplotlibs spec differ to work on the general spec<br />
*** Spec differ can then be used on vtk and other packages in the future <br />
*** Will enable automatic upgrade generation<br />
* [RR] Rework interpreter<br />
** restore abstraction between interpreter/module code<br />
** build looping, streaming, caching into the interpreter<br />
** rework cache<br />
** look into parallelism once basic functionality is there<br />
<br />
=== October 14, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1133 New matplotlib package] (adding numpydoc parser)<br />
<br />
=== October 7, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://github.com/VisTrails/VisTrails/tree/fix-matplotlib-parser New matplotlib parser] (adding numpydoc parser)<br />
** Added numpydoc attribute parser for plots.<br />
*** TODO: Need better port spec reconciliation with call signature parser.<br />
*** We can create a general numpydoc parser, but attribute types are unique to matplotlib.<br />
** Needs package versioning<br />
*** New spec will not support matplotlib < v1.4 due to changed path to axes classes.<br />
*** Load spec version corresponding to installed matplotlib version?<br />
*** Need version downgrades?<br />
<br />
=== September 30, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Updating Matplotlib parser<br />
** Matplotlib docstring parser fails on numpy docstrings<br />
** I have [https://github.com/VisTrails/VisTrails/commit/eae8b0c16260810587e47c41cab53552cbb3cf0a added basic numpydoc parsing]<br />
*** Only used by a few docstrings so far<br />
*** Parsing uses many sources (class tables, signatures, ACCEPTS, method docstrings, definition parsing).<br />
** Caswell said they were thinking about moving to traitlets, but this is not ready yet<br />
<br />
=== September 23, 2015 ===<br />
'''Updates'''<br />
* [TE] VisTrails 2.2.3 released<br />
** Also have pushed to PyPI, binstar, etc.<br />
** Sourceforge vs. GitHub<br />
*** should be able to host releases on GitHub<br />
*** nightly binaries pushed to sf each night<br />
*** old binaries?<br />
* [TE] Re-wrapping MatPlotlib<br />
** Keep static generation - Docstrings are brittle beween matplotlib versions<br />
** Keep generating the executable classes - but create functions, not vistrails Modules <br />
*** Why is there so much patching?<br />
** Re-implemented Alternate PortSpec for InputPortSpec<br />
*** It will now inherit specs from parent (No need to reimplement)<br />
** Module upgrades?<br />
<br />
'''Items to Discuss'''<br />
<br />
=== September 16, 2015 ===<br />
'''Updates'''<br />
* [TE] Job Monitor tests and documentation done<br />
** OK to leave document package in packages directory?<br />
<br />
'''Items to Discuss'''<br />
* [TE] Library wrapping: How to do code patching? [https://docs.google.com/drawings/d/1Sh3eOXot6v2y880jHKQrqXqAwr-0aoqYligg3z8G8MY/edit?usp=sharing Wrapping diagram]<br />
** Insert code into module template (Old VTK code)<br />
** Patch the library that is called (New VTK code)<br />
** Somehow store code in spec and apply when executed?<br />
*** Executed code needs at a minimum access to inputs, outputs, current class<br />
*** Use standard input/output dict and obj reference that the code operates on?<br />
*** Can this be done while keeping the execution abstract?<br />
<br />
* Do new release now?<br />
<br />
=== September 9, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* New release?<br />
** [RR] wants reprounzip<br />
** tej changes & doc<br />
** job monitor & jobmixin fixes<br />
** mongodb<br />
** reprounzip<br />
** warning: MplFigure type is int<br />
** tabledata: add Reshape, DictoToTable & ListToTable, fix ListToTable with numpy arrays<br />
** PythonSource can have same name for an input & output<br />
** don't show spreadsheet at exit<br />
<br />
=== September 2, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Limit autosaves? [https://github.com/VisTrails/VisTrails/pull/1126 #1126]<br />
<br />
=== August 26, 2015 ===<br />
'''Updates'''<br />
* [TE] Added Job support to Parameter Explorations (Requested by Colin), and Mashups<br />
** specify job ids, need to specify different ids for parameter explorations since they have the same version id<br />
** how to deal with parameters passed in on command line<br />
* [General] provenance: should be creating a new version when we execute workflow with changed parameters?<br />
** currently, custom_params annotation stores this in provenance currently<br />
** [RR] API doesn't record provenance if passing in parameters: http://git.io/vsAA5<br />
'''Items to Discuss'''<br />
<br />
=== August 19, 2015 ===<br />
'''Updates'''<br />
* [RR] Improvements to tej, users' guide entry & example [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
* [RR] Internal docs [https://github.com/VisTrails/VisTrails/pull/1116 #1116]<br />
** On ReadTheDocs: [http://vistrails.readthedocs.org/ vistrails.readthedocs.org]<br />
* [TE] Working on Job Monitor documentation<br />
<br />
'''Items to Discuss'''<br />
* [RR] JobMixin and JobMonitor: stable now?<br />
** Definitely needs more tests<br />
* [TE] Stop testing VisTrails 2.0?<br />
<br />
=== August 12, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Internal docs [https://github.com/VisTrails/VisTrails/pull/1116 #1116]<br />
** On ReadTheDocs: [http://vistrails.readthedocs.org/ vistrails.readthedocs.org]<br />
* [RR] JobMixin and JobMonitor: stable now?<br />
** Definitely needs more tests<br />
* [RR] Improvements to tej, users' guide entry & example [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
* [RR] Build broken on Travis; because of IPython 4 released today? ([https://travis-ci.org/VisTrails/VisTrails/builds/75279795 build 992]; [https://github.com/VisTrails/VisTrails/issues/1123 #1123])<br />
<br />
=== August 5, 2015 ===<br />
'''Updates'''<br />
* [TE] Fixes to Jobs<br />
** Could not delete jobs<br />
** Could not run job in group<br />
** Job not reset when calling ModuleSuspended<br />
** Added deleting job from context menu<br />
* [RR] Writing documentation for everything [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
<br />
'''Items to Discuss'''<br />
<br />
=== July 29, 2015 ===<br />
'''Updates'''<br />
* RR still looking into new interpreter thing<br />
** Goal is to take out scheduling logic from Module so it can be split in multiple processes, and so that smarter strategies can be added in time<br />
** This means some work on packages<br />
** Spreadsheet can live in kernel process? Still some UI stuff that will take work (changing configuration, persistent archive's viewer, ...)<br />
* [TE] Problems running examples [https://github.com/VisTrails/VisTrails/pull/1111 #1111]<br />
** Testing of more examples requires additional packages on the test machines.<br />
** Fixed faulty line-ending in PythonSource:s failing on Python 2.6.<br />
** Test suite now testing SUDSWebServices (If web service is down, test suite will fail)<br />
** preferences.py test failed reloading 'dialogs' package, switched to using 'URL'.<br />
<br />
'''Items to Discuss'''<br />
<br />
=== July 22, 2015 ===<br />
'''Updates'''<br />
* [TE] mailing lists back online<br />
* [TE] Working on [https://github.com/VisTrails/VisTrails/issues/1107 #1107]<br />
* [RR] Working on ReproZip package <br />
<br />
'''Items to Discuss'''<br />
* [RR] MongoDB package [https://github.com/VisTrails/VisTrails/pull/1106 #1106]<br />
* [RR] Example for tej docs? [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
<br />
=== July 15, 2015 ===<br />
'''Updates'''<br />
* 2.2.2 released<br />
<br />
'''Items to Discuss'''<br />
* RR is considering executing everything in an IPython kernel (i.e. separate Python interpreter, like the one spawned for a notebook)<br />
** It's a separate process that we can restart without restarting the app/gui<br />
** We can isolate the execution environment (e.g. for the server)<br />
** We can run the whole thing remotely (if your data is elsewhere, just run VisTrails locally and the pipeline elsewhere)<br />
** Pipeline execution no longer makes the interface hang, it just makes the kernel hang (but that's fine)<br />
** We can use notebooks as modules (probably way nicer than the PythonSource module)<br />
** We can run multiple kernels so long as the ports carry things that are serializable<br />
*** meaning we can put the multithreaded-interpreter without all the hacky parts it has now<br />
*** we can run IPython kernels in all the languages IPython supports, [https://github.com/ipython/ipython/wiki/IPython-kernels-for-other-languages currently 46]<br />
<br />
=== July 8, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Buildbot github hook not working after IP address change<br />
* [TE] No reference to VistrailsApplication ([https://github.com/VisTrails/VisTrails/issues/1103 #1103])<br />
** [RR] Added PR [https://github.com/VisTrails/VisTrails/pull/1104 #1104]<br />
* [TE] Reopening VT file after saving with bundled subworkflow won't offer subworkflow upgrade ([https://github.com/VisTrails/VisTrails/issues/1102 #1102])<br />
** allow manual delete to fix right now<br />
** fix this on top of the use-uuid branch<br />
* [TE] Release VisTrails 2.2.1? ([https://github.com/VisTrails/VisTrails/blob/v2.2/CHANGELOG CHANGELOG])<br />
** [https://github.com/VisTrails/VisTrails/pull/1093 output-modules-use-global-dir]<br />
** [https://github.com/VisTrails/VisTrails/milestones/version%202.2 Current issues for 2.2]<br />
<br />
=== July 1, 2015 ===<br />
'''Updates'''<br />
* [TE] PROV fixed<br />
* [TE] Working on subworkflow issues<br />
* [RR] Considering reworking the controller (log vs exception problem, retained upgrades causing interferences) and also the interpreter (IPython?)<br />
<br />
'''Items to Discuss'''<br />
<br />
=== June 24, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* UV-CDAT<br />
* [TE] Can a cyclic workflow be valid? ([https://github.com/VisTrails/VisTrails/issues/1097 #1097])<br />
** focus on disabling the ability to create cyclic pipelines because more things break than just this with a cyclic pipeline<br />
* [RR] Relative paths ([https://github.com/VisTrails/VisTrails/pull/1057 #1057])<br />
** This interacts with the new bundle; how do handle packing files inside the VT bundle?<br />
<br />
=== June 17, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] current_version and reusing existing upgrades are broken ([https://github.com/VisTrails/VisTrails/issues/1095 ticket #1095])<br />
** Could be that export to PROV is using an unflushed upgrade pipeline?<br />
** current_version would then be correct.<br />
** It may work to flush the actions before exporting?<br />
<br />
=== June 10, 2015 ===<br />
'''Updates'''<br />
* DAT: fixed VTK issue on Linux and Mac<br />
* Still crashes on Windows. Need help! Reminder: this works in the VisTrails spreadsheet (QCellPresenter), although no widget get changed there during a drag<br />
** Is it a VTK bug?<br />
** Is it simply impossible to change widgets during the drag, should we do it a different way?<br />
** Did I miss something that is done in VisTrails but somehow not in DAT?<br />
** Low prio, UV-CDAT doesn't run on Windows anyway<br />
<br />
'''Items to Discuss'''<br />
* [TE] Executable Paper ([https://github.com/VisTrails/VisTrails/issues/1086 ticket #1088] [https://github.com/VisTrails/VisTrails/pull/1088 Updated pull request])<br />
* Single instance code and batch mode<br />
<br />
=== June 3, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Executable Paper ([https://github.com/VisTrails/VisTrails/issues/1086 ticket #1088] [https://github.com/VisTrails/VisTrails/pull/1088 pr])<br />
** Requires fixes to command line parameters, Output modules, and batch mode<br />
** How to test this<br />
** Updated missing/outdated flags<br />
** Fixed view issues when generating graphs<br />
* [TE] batch mode<br />
** SpreadsheetOutput not enabled in batch mode. Should we check is_running_gui instead?<br />
** Other instance setting flags from caller<br />
** Is graphsAsPdf replacing spreadsheetDumpPdf?<br />
** graphsAsPdf true by Default?<br />
** Batch mode executing by default (Not needed when generating graphs)<br />
** Re-added workflowInfo as withWorkflowInfo for writing graph and xml workflow<br />
** [DK] batch mode should be outputting to files or stdout, shouldn't always trigger SpreadsheetMode<br />
** execute flag, maybe make execution the default and allow a "--no-execute" if you only want to capture graphs, for example<br />
<br />
=== May 27, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] API question<br />
** http://lists.vistrails.org/pipermail/vistrails-users/2015-May/000357.html<br />
* Status of DAT<br />
** Can add VCS plot to DAT but need to work on configuration windows<br />
** Documentation for graphics templates for UV-CDAT/vcs?<br />
* Qt support?<br />
<br />
=== May 20, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Reviving the DAT, integrating scripting & porting UV-CDAT!<br />
** On [https://github.com/VisTrails/DAT GitHub] ([https://github.com/VisTrails/DAT/issues issues])<br />
** Merging 2 years of development taking longer than expected, but getting there. The plan is to get the patches in VisTrails and never fork again, we never want to get in UV-CDAT's situation (and don't need to).<br />
** VTK cell works fine on Linux but there was flickering on Mac & Windows before; still issues on Mac (Windows status unknown)<br />
** Can get a VCS plot soon (but will need VTK cell fix)<br />
** Integrate in UV-CDAT's build system (so we have cdms, VCS, ...) -> RR can do this, low priority<br />
** How do we integrate scripting?<br />
*** We want to be able to seamlessly make changes to a plot by changing Python code<br />
*** Define new plots by entering Python code without writing modules/packages?<br />
* [RR] UV-CDAT [https://github.com/UV-CDAT/uvcdat/issues/1275 needs] [https://github.com/VisTrails/VisTrails/pull/1073 #1073], please take a look<br />
* [RR] Subworkflow issues<br />
** [https://github.com/VisTrails/VisTrails/issues/1065 #1065]: missing attribute (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/1066 #1066]: groups in subworkflows were ignored when finding dependencies (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/725 #725]: "upgrade subworkflow" button not popping up (fixed); also, namespace issue<br />
** [https://github.com/VisTrails/VisTrails/issues/1071 #1071]: subworkflows don't go through upgrades<br />
* [RR] [https://github.com/VisTrails/VisTrails/issues/1074 #1074]: can't load and edit a single pipeline if upgrade happen<br />
<br />
=== May 13, 2015 ===<br />
'''Updates'''<br />
* [RR] Test skipping whitelist ([https://github.com/VisTrails/VisTrails/pull/1069 #1069]) -- low priority<br />
* [RR] Custom matplotlib modules can't be compatible with both 2.1 and 2.2 ([https://github.com/VisTrails/VisTrails/issues/1067 #1067]); should be fixed for ALPS ([https://github.com/VisTrails/VisTrails/issues/1070 #1070])<br />
<br />
'''Items to Discuss'''<br />
* [RR] UV-CDAT [https://github.com/UV-CDAT/uvcdat/issues/1275 needs] [https://github.com/VisTrails/VisTrails/pull/1073 #1073], please take a look<br />
* [RR] Subworkflow issues<br />
** [https://github.com/VisTrails/VisTrails/issues/1065 #1065]: missing attribute (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/1066 #1066]: groups in subworkflows were ignored when finding dependencies (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/725 #725]: "upgrade subworkflow" button not popping up (fixed); also, namespace issue<br />
** [https://github.com/VisTrails/VisTrails/issues/1071 #1071]: subworkflows don't go through upgrades<br />
* [RR] [https://github.com/VisTrails/VisTrails/issues/1074 #1074]: can't load and edit a single pipeline if upgrade happen<br />
<br />
=== May 7, 2015 ===<br />
'''Updates'''<br />
* [RR] Export/import workflow to Python working!<br />
<br />
'''Items to Discuss'''<br />
* [Claudio] UV-CDAT<br />
** The UV-CDAT project is the biggest user base of VisTrails<br />
** VisTrails package management provides a lot of friction towards people plugging in their code<br />
** Need to make it easy to integrate your random Python scripts in the system without having to deal with all the boilerplate, at least in the first step<br />
** [RR] argues that modules are still good; UV-CDAT shouldn't move towards a purely script-based backend<br />
** [RR] export/import with Python could reduce a lot of that friction by allowing 1) to edit workflow as Python 2) to open up boxes automatically if needed code doesn't match actual modules<br />
** ...<br />
<br />
=== April 29, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] <strike>Abstractions</strike> subworkflows status ([https://github.com/VisTrails/VisTrails/labels/subworkflows tickets])<br />
* [RR] matplotlib compatibility (2.1 & 2.2), [https://github.com/VisTrails/VisTrails/issues/1067 #1067]<br />
** RR to try and fix ALPS matplotlib modules<br />
<br />
=== April 22, 2015 ===<br />
'''Updates'''<br />
* 2.2.0 has been released!<br />
* Windows issue (via email from Ryan)<br />
** issue with manifest file (may be a new file in VTK6?)<br />
** Tommy has regenerated new Windows builds<br />
* Binaries, pypi, and conda released<br />
* [RR] Export as script<br />
* Python sources using VTK need to switch to SetInputData (users should be aware of this)<br />
<br />
=== April 15, 2015 ===<br />
'''Updates'''<br />
* Ready for 2.2.0 (apart from binary/deps issues)<br />
** Missing some libs (scikit-learn, tej, tdparser, SQLAlchemy+connectors<br />
** Windows: runvistrails.py is no longer used, so the PATH is wrong<br />
** Windows: pip is broken, but it probably wouldn't work anyway because of permission issues (disable this?)<br />
* Queries, upgrades and getPipeline() usage ([https://github.com/VisTrails/VisTrails/issues/1054 #1054])<br />
** Getting a pipeline with getPipeline() is not safe: it might return an invalid pipeline<br />
** This is used in many places throughout the code, like queries<br />
** Upgrading would require going through the controller, but that creates new actions<br />
* [TC] Avoid copying a module's output if it's used as input by exactly one downstream module ([https://github.com/VisTrails/VisTrails/issues/1060 #1060]) (useful for big numpy arrays you can update in-place)<br />
<br />
=== April 8, 2015 ===<br />
'''Updates'''<br />
* [DK] Merged RR's changes for output modules (1012 and 1013)<br />
** RR will merge remaining changes, then create v2.2 branch!<br />
<br />
'''Items to Discuss'''<br />
* BNL need numpy array to VTK image<br />
** Looks like VTK has some helpers for this<br />
** We will help if issues arise<br />
** Will contribute back to VisTrails package<br />
* Upgrade issue: [https://github.com/VisTrails/VisTrails/issues/1017 #1017]<br />
** Automatic upgrades should happen between versions of provided upgrades<br />
** Our existing upgrades work around this so it doesn't need to be 2.2.0<br />
* Corner-case VTK modules<br />
** No longer need VTKCell input port, so don't interfere with registry and API anymore<br />
** Still work weirdly, people probably shouldn't use them<br />
** But we have lots of clunky modules since we wrap the whole of VTK; some people might rely on this and know how to use them, let's keep them anyway<br />
** Ready for 2.2.0<br />
<br />
=== April 1, 2015 ===<br />
'''Updates'''<br />
* [RR] UV-CDAT: bugfixing for 2.2, long-term plans: implement scripting import/export in VisTrails, port to UV-CDAT<br />
* Possibly, try to move to regular VisTrails to use new features<br />
<br />
'''Items to Discuss'''<br />
* 2.2 release: nothing much is pending anymore, release next week?<br />
** [TE] vtkExporter classes [https://github.com/VisTrails/VisTrails/issues/1032 #1032]<br />
** [RR] Hiding upgrades in version tree might make it (or might be 2.2.1 so we can test it out) [https://github.com/VisTrails/VisTrails/issues/949 #949]<br />
** Output modules changes to go in<br />
* add note to documentation about order of parameters in VTK<br />
* add issue about exporter upgrades if not already there<br />
<br />
=== March 25, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* 2.2 release<br />
** Checklist on Github: https://github.com/VisTrails/VisTrails/wiki/2.2-Release-Checklist<br />
** Review output modules<br />
** RR has a couple more issues to fix<br />
** Ready to go -- sign app package for OSX? [https://github.com/VisTrails/VisTrails/issues/984 #984] We need access to the Apple Membership team<br />
<br />
=== March 18, 2015 ===<br />
'''Updates'''<br />
* [TE] VTK6 works<br />
<br />
'''Items to Discuss'''<br />
* [RR] Release v2.1.5 with backported tabledata?<br />
** MTA example needs updated tabledata (for JoinTables)<br />
** Google Maps package still not available<br />
* [RR] Work torward v2.2.0?<br />
** changes:<br />
*** new persistence<br />
*** API changes<br />
*** output module changes (upgrades?), maintain cells but try to upgrade<br />
*** not wrapping stuff<br />
*** VTK6? yes<br />
*** JobSubmission stuff?<br />
*** relabeling for upgrades [https://github.com/VisTrails/VisTrails/issues/949 #949]<br />
** makes sense, needs the tree view code to be updated, check selection<br />
** See [https://github.com/VisTrails/VisTrails/wiki/2.2-Release-Checklist 2.2 checklist]<br />
* Discussion of [https://github.com/VisTrails/VisTrails/issues/1016 #1016]<br />
** plumbing between outputs and output modes, how to define a mode that works for many outputs without writing for each output?<br />
<br />
=== March 11, 2015 ===<br />
'''Updates'''<br />
* [RR] persistent_archive done; merge? ([https://github.com/VisTrails/VisTrails/pull/755 #755])<br />
** note about the focus events for widgets<br />
** TE be aware of file_archive for future binaries that include persistent_archive<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/998 New VTK package] finished<br />
<br />
'''Items to Discuss'''<br />
* [RR] Unmark UV-CDAT/VisTrails as a fork of VisTrails/VisTrails? (see last week; decision needed)<br />
** RR email to JF about this<br />
** yes; email sent to Github<br />
* [RR] Switching order of output ports ([https://github.com/VisTrails/VisTrails/pull/1006 #1006])<br />
** added port specs are sorted at a separate spot (Module.*_port_specs properties) than those in the registry (which are sorted in the registry), but those two lists are just combined without respect to sort keys<br />
** need to determine whether the two lists should be merged or remain distinct<br />
** should make sure that order of input ports and output ports makes parallel connections for things with same order<br />
** DK suggests breaking backward compatibility here: workflows still run, can fix easily if a problem in an existing package.<br />
* [RR] Question about output modes ([https://github.com/VisTrails/VisTrails/pull/1007 #1007]), how to integrate in API ([https://github.com/VisTrails/VisTrails/pull/24 #24])<br />
** Should ImageFileMode be removed? ("image" is not a mode, "file" is)<br />
** ImageOutput missing?<br />
** Feel free to change how formats works<br />
* [TE] Test suite segfaults on Fedora 17 virtualbox. Install newer version? (Support ended in 2013)<br />
<br />
=== March 4, 2015 ===<br />
'''Updates'''<br />
* [TE] VTK wrapper<br />
** Works on VTK 5.10<br />
** Still need to test VTK 6<br />
** New general wrappers for python functions and classes <br />
<br />
'''Items to Discuss'''<br />
* [RR] Unmark UV-CDAT/VisTrails as a fork of VisTrails/VisTrails?<br />
** When [http://i.imgur.com/UTY2qfg.png filing a pull request] from UV-CDAT/VisTrails, VisTrails/VisTrails is selected by default<br />
** People keep forgetting to change the default ([https://github.com/VisTrails/VisTrails/pull/956 #956] [https://github.com/VisTrails/VisTrails/pull/968 #968] [https://github.com/VisTrails/VisTrails/pull/999 #999] [https://github.com/VisTrails/VisTrails/pull/1000 #1000] [https://github.com/VisTrails/VisTrails/pull/1003 #1003] [https://github.com/VisTrails/VisTrails/pull/1004 #1004] [https://github.com/VisTrails/VisTrails/pull/1005 #1005])<br />
** Only way to change that is to not have it marked as a fork of VisTrails/VisTrails<br />
** Github staff [https://gist.github.com/remram44/b364c24ef1e462ce6e2f can make that change for us]; should we do it?<br />
** Juliana: comments on this? visibility vs. convenience/annoyance for developers<br />
* [RR] What about [https://github.com/VisTrails/VisTrails/issues/991 #991]?<br />
<br />
=== February 25, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] vtkviewcell for infovis support, can we unify with VTKCell?<br />
** need to test this<br />
* [TE] vtk wrapping<br />
** Mostly finished<br />
** VTK 5.10 produce incorrect results with old wrapping<br />
*** Old wrapper is based mostly on VTK 4<br />
*** Most vtk_examples affected<br />
*** [http://www.vtk.org/Wiki/VTK/VTK_6_Migration/Replacement_of_SetInput]<br />
*** should be able to upgrade from SetInput to SetInputData (need to drop GetOutput and replace with self ports)<br />
*** can we change vtkInstance to just return self and not wrap things<br />
** terminator example not working under 5.8?<br />
* How does VTK wrapping fit into general wrapping framework?<br />
* [RR] new persistence package<br />
<br />
=== February 18, 2015 ===<br />
'''Updates'''<br />
* [RR] New VisTrails API and IPython integration ([https://github.com/VisTrails/VisTrails/pull/24 #24])<br />
<br />
'''Items to Discuss'''<br />
* [TE] VTK wrapping<br />
** [https://gist.github.com/rexissimus/16b74693545b5e7c1035 Benchmarking vtk package]<br />
*** Old: 24.7 seconds <br />
*** New: 10.5 seconds (Except first time that adds 8 sec)<br />
**** The parsing that calls is_abstract (that tries to instanciate all vtk classes) is now only run the first time.<br />
**** get_items_from_sigstring takes 2 seconds, maybe we can use a lookup dict for already computed sigstrings?<br />
** Now using a general python function wrapper<br />
*** VTK classes are wrapped into python function that does not depend on vistrails<br />
*** VTK functions can be executed without vistrails<br />
*** The spec maps functions into vistrails modules, but can also describe wrapping <br />
*** A general python function wrapper that supports <br />
**** kwarg inputs<br />
**** single, list, dict outputs<br />
**** callback for progress reporting<br />
**** temporary file generator for using FilePool<br />
**** optional output generation<br />
*** Creating specs:<br />
***** Create spec by hand<br />
***** Auto-create spec outline (TODO) and manually finish it<br />
***** Dynamically create spec (VTK)<br />
***** Implement documentation wrappers (Can use scikitlearn wrapper to wrap numpydoc) (TODO)<br />
***** Classes as bad functions needs to be wrapped in new functions before they are wrapped. This is different for each package.<br />
****** Classes is hard: Like VTK, and matplotlib. Scikit-learn does still not wrap classes<br />
***** Spec diffing and patching could be done using code from matplotlib.<br />
** Still needs upgrades from old VTK package<br />
*** Is it possible to dynamically wrap functions, e.g, you see a SetFunc and just remove the 'Set' prefix. Or do you need to create a complete mapping?<br />
<br />
=== February 11, 2015 ===<br />
'''Updates'''<br />
* Update from Friday's meeting<br />
** discussed VisTrails internals<br />
** discussed wrapping<br />
*** xml discussion, hard to modify because tied to db code<br />
*** TE has made it possible to add the schema-defined attributes to the intermediate representation<br />
*** higher-level operations on the port specs<br />
** make sure the simple case works<br />
*** [JF] take a simple package with documentation and figure out what the base case for wrapping is<br />
<br />
'''Items to Discuss'''<br />
* [TE] VTK wrapping<br />
** Dynamic loading works<br />
** Reading XML is fast enough, but [https://gist.github.com/rexissimus/8272d3c776ccfa115d82 unserializing data is slow]<br />
** Working on patterns for patching<br />
** Matplotlib has many advanced patterns like argument ordering, nested arguments, alternateSpecs, output types.<br />
*** Having all this in a general wrapper might confuse users?<br />
** [RR] Delay module (except for identifiers) until you need it---e.g. don't deal with port specs, etc. until necessary<br />
* Scripting Support [https://github.com/VisTrails/VisTrails/pull/950 #950]<br />
** [RR] Issue with getting code from modules<br />
** Design a simple solution<br />
** [JF] Couldn't you use modules as black boxes without conversion, just to call into modules/subworkflows easily from e.g. IPython?<br />
*** [RR] This is a job for the API, and a very separate use case. See [https://github.com/VisTrails/VisTrails/pull/24 #24]<br />
<br />
=== February 4, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* Wrapping<br />
** Format to use? Currently XML (like current matplotlib)<br />
*** JSON and YAML have simple "to python dictionary" methods<br />
*** But don't stream<br />
*** YAML a lot easier for humans<br />
** [DK] vtk-new-package also changes parameter names, creates enumerations<br />
*** intermediate schema needs to be extensible<br />
*** packages will want to store there specific infos for compute() method generation<br />
*** also might have specs-altering info, like matplotlib's alternateSpec<br />
** representation to code , registry already has schema for some aspects<br />
** [RR] We might want to see if Module subclasses can be created lazily<br />
*** no need to create all the classes just to register them in the registry and never actually use most of them<br />
*** future effort<br />
* [RR] Where should VisTrails packages live?<br />
** tej installs as 'vistrailspkg.tej', TE installed it as 'userpackages.tej'<br />
** Currently, standard packages are 'vistrails.packages.', user packages are 'userpackages.' and packages loaded through pkg_resources might be anything<br />
** [RR] Use 'vistrailspkg.' everywhere?<br />
** Long-term effort to simplify package distribution/installation (and have VisTrails get them automatically?)<br />
<br />
=== January 28, 2015 ===<br />
'''Updates'''<br />
* T. Caswell to come visit on Fri 6 to discuss wrapping work<br />
<br />
'''Items to Discuss'''<br />
* [TE] New VTK wrapping<br />
** Current code by DK seems a good deal faster<br />
** Generates XML that can be patched/tweaked, generates Python code from it, VisTrails only loads generated Python code<br />
** RR would rather have VisTrails load intermediate representation (= XML) directly, wants to make sure this is not slower<br />
** The goal is to turn the intermediate step into something generic that would be used for every wrapped package (vtk, numpy, matplotlib, sklearn, java) instead of each having its own<br />
** TC has its own code at [https://github.com/Nikea/VTTools/blob/master/vttools/wrap_lib.py github:VTTools] which parses numpy docstrings and generates modules, doesn't yet handle classes or persist anything<br />
* Web crawler<br />
** Right now, TE starts jobs for "start crawler", "stop crawler", "install classifier"<br />
** RR would rather have the crawling be a job as far as VisTrails and tej are concerned<br />
** The whole thing would be one pipeline: load examples, train classifier, start crawler [check for job, kill previous one, upload model, start processes], get snapshot, visualize<br />
*** Need some support in tej and job submission system: long-running jobs, stop a job (wait for it to finish?), restart a job even though results are cached<br />
<br />
=== January 21, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Unified wrapping method discussion [https://github.com/VisTrails/VisTrails/issues/991 #991]<br />
** TE to work on reusable method with intermediate representation, starting with VTK<br />
* [RR] Examples for scikit-learn: JF has an old example using Weka with parameter exploration (not currently in source tree)<br />
** AM's examples are enough<br />
* [AM] scikit-learn package is done, merge it in? [https://github.com/VisTrails/VisTrails/pull/955#issuecomment-70286788 #955]<br />
** RR will merge<br />
* [RR] What should copyright headers say? [https://github.com/VisTrails/VisTrails/pull/994#issuecomment-70173935 #994]<br />
** Let's keep everything in there: Utah/Poly/NYU<br />
<br />
=== January 14, 2015 ===<br />
'''Updates'''<br />
* [TE] Working on classifier<br />
* [RR] Scripting integration, work in progress<br />
<br />
'''Items to Discuss'''<br />
* [RR] Unified wrapping method discussion ([https://github.com/VisTrails/VisTrails/issues/991 #991])<br />
** Let's talk next week, [AM] and [DK] are not here<br />
<br />
=== January 7, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* make sure that we address critical issues, questions, and pending review branches in a timely manner<br />
* scripting support<br />
** [RR] no issues if we want to just keep annotations in the generated code to allow the link back to a workflow<br />
** [RR] can translate from workflow to script, working on script to workflow<br />
** will work for parameter value changes, structural changes require changes to the annotations<br />
** need to publish best practices here<br />
** would be cool to do looping in scripts (easier interface than with workflows)<br />
* notebook support (convert form notebook to workflow)<br />
** RR will sync with FC on this<br />
* Issue with console in built-from-scratch<br />
** [TC] iPython rearranged some of the completion stuff in 2.2 and 2.3<br />
** binary has old version of iPython -> 1.0.0, should we update?<br />
* [TC] automated wrapping of numpy and scipy<br />
** discovered a bunch of malformed documentation in numpy and scipy<br />
** has github repo for vistrails tools<br />
** example modules wrap a bunch of R stuff (not baked in, just how things are)<br />
** will be pushing wrapping logic up<br />
** port names forbidden (window and domain)<br />
** have an import hook to get from yaml directly to VisTrails Modules<br />
** should work for any python modules with well-formed numpy docstrings.<br />
* [Action] should make it clear in documentation that Constant now means serializable not that the value doesn't change (e.g. List)<br />
* [TC] might be interesting to try to build components of matplotlib and accumulate in figure (long-term project, but thinking about how this might work)<br />
* [TE] build and build scripts<br />
** completely automatic, buildbot<br />
** need to set the build machines for the environment we want for the binary<br />
** would virtualenv work here?<br />
** [TC] anaconda can pin versions, potential path to test different configurations<br />
** Q: upload nightly binary builds? A: makes sense, make sure they are well-labeled<br />
* sourceforge stats: e.g. http://sourceforge.net/projects/vistrails/files/vistrails/nightly/vistrails-src-nightly.tar.gz/stats/timeline?dates=2014-01-01+to+2015-01-07<br />
* package issues (see Remi's message)<br />
* [TE] Scope of tej<br />
** Support single ssh commands?<br />
** Queue can be used as a remote machine (crawler is using queue.call*)<br />
* SourceForge stats: http://sourceforge.net/projects/vistrails/files/vistrails/nightly/vistrails-src-nightly.tar.gz/stats/timeline?dates=2014-01-01+to+2015-01-07<br />
<br />
== Older meetings ==<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=11513Development2016-04-06T14:50:24Z<p>Remi: /* April 6, 2016 */</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2016 ==<br />
<br />
=== April 6, 2016 ===<br />
'''Updates'''<br />
* [TE] Query View improvements<br />
** Done except for displaying "Version Info" and #1164<br />
* [RR] Working on finishing new interpreter (finally!)<br />
<br />
'''Items to Discuss'''<br />
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])<br />
** Try latest upgrade first, then previous.<br />
** If all fail, select latest upgrade?<br />
<br />
=== March 30, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
<br />
* [TE] Query View improvements<br />
** Unified do_version_switch and get_upgrades as validate_version()<br />
** Problems with query view using its own controller<br />
*** Fixed module info not being displayed for query results<br />
*** Still no version info for query results<br />
*** Still an issue with version query view not being updated<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== 2015 ==<br />
<br />
=== December 30, 2015 ===<br />
'''Updates'''<br />
* Bokeh package finished<br />
** Can output to Browser/IPython/Spreadsheet (Limited functionality in spreadsheet)<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Bokeh needs QWebEngine (Chromium) in Qt5 which requires PyQt5<br />
*** Many plotting libraries only work in browsers<br />
*** But Chromium may be a big and controversial dependency<br />
*** Or create a browser version of the spreadsheet?<br />
** Some users have requested Python 3<br />
** Not backwards compatible?<br />
** Target for VisTrails 3.0 with new interpreter?<br />
<br />
=== December 16, 2015 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Users list question<br />
* [TE] Wrapping Bokeh<br />
** Added bokeh property parser<br />
*** complements numpydoc parser<br />
*** Not all attributes are available, partially because I am using a pre-release version.<br />
** Wrapped modules<br />
*** bokeh.plotting finished (Backwards like matplotlib: Create plot then draw on it)<br />
*** bokeh.charts mostly done (Missing arguments)<br />
** QWebView mostly works (but no zooming or buttons)<br />
*** It would be better to use external browser or QWebEngineView<br />
** Added improvements to wrapper<br />
*** Generalized port translations (Color/Path to native types)<br />
*** Can use multiple docstring/property parsers simultaneously<br />
<br />
=== December 9, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Looking into wrapper<br />
** Will port TensorFlow<br />
** Doing a pass on the wrapper code & doc<br />
*** UX improvements<br />
** Factorize high-level docstring parsers into core.wrapper? (sphinx, numpy, google)<br />
** Problem with name vs module_name (fails sklearn tests because class_by_name() is broken)<br />
** Looking into compressing specs, lazy registration of modules<br />
<br />
=== December 2, 2015 ===<br />
'''Updates'''<br />
* [TE] Package wrapping<br />
** Wrapped most of numpy/scipy using __all__<br />
** Added class attribute and method access for classes (Can add to class or to separate inspector module)<br />
** TODO: Document wrapping procedure<br />
<br />
'''Items to Discuss'''<br />
<br />
=== November 25, 2015 ===<br />
'''Updates'''<br />
* [RR] New interpreter is coming along<br />
** Everything is a stream<br />
** Task system, eventually work-stealing parallelism<br />
** Streams no longer advance in lock-step<br />
*** Allows for constructions like filter, join, sorted-merge<br />
** depth>1 still to be tested, no current plan to allow this through the module interface<br />
*** Through looping though, can do streams of streams<br />
<br />
'''Items to Discuss'''<br />
* [TE] Package wrapping<br />
** Wrapped most numpy functions (no polynomials)<br />
** and half of scipy classes and functions<br />
** Added functions manually from documentation (no parseable lists?)<br />
** Focusing on data flow constructs (no property/method access for classes)<br />
*** Using spec diff (empty for now) and dynamic parser (stores spec in .vistrails/)<br />
** Python class/function wrapper now stable<br />
** Add google doc parser?<br />
*** Is there a parser available?<br />
<br />
=== November 18, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Package wrapping<br />
** Numpy class wrapping works<br />
*** Basic type is List, since most types are array_likes.<br />
*** TODO: More classes and functions<br />
*** Some operations are in-place and some aren't, and docstring not super clear<br />
*** [RR] It seems only methods mutate, and most have a numpy.xx function equivalent; just manually go through the methods and remove the mutating ones?<br />
** Class wrapper is modular<br />
*** Docstring parser and type string parser can be customized<br />
*** Classes can have optional attribute/method ports<br />
**** Inspectors and attribute/method modules can be created separately<br />
** Use PythonCalc as an example<br />
*** Can be re-implemented as a function with a parseable docstring<br />
*** [RR] PythonCalc is just an example, and it might actually make sense to get rid of it, or do a proper math package with scalar operations (as separate modules, no combobox)<br />
** [RR] TensorFlow to use wrapping as well<br />
*** Very simple (only types 'tensor' & 'variable')<br />
<br />
=== November 11, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Package wrapping<br />
** Wrapping numpy's classes using numpydoc (ndarray)<br />
*** Wrapping constructor arguments, attribute getters/setters as module ports<br />
**** Then how to access attributes afterwards?<br />
***** Use input value port to class modules?<br />
*** Wrap methods as modules? E.g., `ndarray.shape`.<br />
*** Function wrapping is almost a subset of class wrapping<br />
**** May be able to use the same parsing/execution methods<br />
** VTK's non-getter/setter methods could be wrapped as modules<br />
*** We could then remove extra logic in interpreter for keeping function order<br />
** Bokeh uses autogenerated docstrings<br />
*** We may be able to read the specification directly<br />
<br />
* [RR] [https://github.com/VisTrails/VisTrails/pull/1138 TensorFlow package]<br />
** Basic setup working, can execute the Mandelbrot example<br />
** Will autogenerate the operations<br />
<br />
=== November 4, 2015 ===<br />
'''Updates'''<br />
* [RR] Interpreter work requires a fix for [https://github.com/VisTrails/VisTrails/issues/1137 DB issue #1137]<br />
* [DK] Kitware's Resonant [http://www.kitware.com/source/home/post/173]<br />
** [https://github.com/girder/girder Girder]: data management system<br />
** [https://github.com/Kitware/romanesco Romanesco]: execution engine, uses [http://www.celeryproject.org/ Celery] for task management<br />
** Resonant Flow: web app for editing and executing workflows<br />
<br />
'''Items to Discuss'''<br />
* [TE] Added OSX Lion (10.7) VM on build machine using vagrant (seems ok with license?)<br />
** New builds works on Lion<br />
** Change the minimum requirement to 10.7?<br />
* [TE] Package wrapping<br />
* Added upgrade suggestions using 2 spec versions ([http://pastebin.com/30XUU8Gp example])<br />
** using name edit distance to find matches<br />
** Could use something better like port similarity for modules, and type similarity for ports<br />
** Should write python upgrade code?<br />
** Added spec to sklearn package<br />
*** Can now diff spec versions and keep the spec static<br />
*** There will be problems with downgrades and package versions<br />
<br />
=== October 29, 2015 ===<br />
'''Updates'''<br />
* [TE] nightly-build-mac fixed (Needed to approve xcode license)<br />
<br />
'''Items to Discuss'''<br />
* [TE] matplotlib wrapper<br />
** ported to general spec<br />
** Diff can be used on general spec<br />
*** Only needed minor changes<br />
*** Fixed indexing bug that corrupted specs<br />
*** Can now move diff tools to core/wrapper<br />
** Add new plots?<br />
*** New plots in mpl 1.3: eventplot<br />
*** New plots in mpl 1.4: angle_spectrum, magnitude_spectrum, phase_spectrum', violinplot<br />
*** No new plots in mpl 1.5 (from looking at boilerplate.py)<br />
<br />
=== October 21, 2015 ===<br />
'''Updates'''<br />
* [RR] Alexis has arrived, will be working with [RR] on a more efficient interpreter<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1133 New matplotlib package]<br />
** Supporting multiple package versions<br />
*** Check which version can be loaded [https://github.com/VisTrails/VisTrails/pull/1135 #1135]<br />
*** We cannot show version requirements in the list of packages because old packages are loaded when importing the codepath<br />
** Visual diff does not work well, but we can use the spec differ to see differences.<br />
** Unifying vtk and matplotlib wrapping specs (WIP)<br />
*** And porting matplotlibs spec differ to work on the general spec<br />
*** Spec differ can then be used on vtk and other packages in the future <br />
*** Will enable automatic upgrade generation<br />
* [RR] Rework interpreter<br />
** restore abstraction between interpreter/module code<br />
** build looping, streaming, caching into the interpreter<br />
** rework cache<br />
** look into parallelism once basic functionality is there<br />
<br />
=== October 14, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1133 New matplotlib package] (adding numpydoc parser)<br />
<br />
=== October 7, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://github.com/VisTrails/VisTrails/tree/fix-matplotlib-parser New matplotlib parser] (adding numpydoc parser)<br />
** Added numpydoc attribute parser for plots.<br />
*** TODO: Need better port spec reconciliation with call signature parser.<br />
*** We can create a general numpydoc parser, but attribute types are unique to matplotlib.<br />
** Needs package versioning<br />
*** New spec will not support matplotlib < v1.4 due to changed path to axes classes.<br />
*** Load spec version corresponding to installed matplotlib version?<br />
*** Need version downgrades?<br />
<br />
=== September 30, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Updating Matplotlib parser<br />
** Matplotlib docstring parser fails on numpy docstrings<br />
** I have [https://github.com/VisTrails/VisTrails/commit/eae8b0c16260810587e47c41cab53552cbb3cf0a added basic numpydoc parsing]<br />
*** Only used by a few docstrings so far<br />
*** Parsing uses many sources (class tables, signatures, ACCEPTS, method docstrings, definition parsing).<br />
** Caswell said they were thinking about moving to traitlets, but this is not ready yet<br />
<br />
=== September 23, 2015 ===<br />
'''Updates'''<br />
* [TE] VisTrails 2.2.3 released<br />
** Also have pushed to PyPI, binstar, etc.<br />
** Sourceforge vs. GitHub<br />
*** should be able to host releases on GitHub<br />
*** nightly binaries pushed to sf each night<br />
*** old binaries?<br />
* [TE] Re-wrapping MatPlotlib<br />
** Keep static generation - Docstrings are brittle beween matplotlib versions<br />
** Keep generating the executable classes - but create functions, not vistrails Modules <br />
*** Why is there so much patching?<br />
** Re-implemented Alternate PortSpec for InputPortSpec<br />
*** It will now inherit specs from parent (No need to reimplement)<br />
** Module upgrades?<br />
<br />
'''Items to Discuss'''<br />
<br />
=== September 16, 2015 ===<br />
'''Updates'''<br />
* [TE] Job Monitor tests and documentation done<br />
** OK to leave document package in packages directory?<br />
<br />
'''Items to Discuss'''<br />
* [TE] Library wrapping: How to do code patching? [https://docs.google.com/drawings/d/1Sh3eOXot6v2y880jHKQrqXqAwr-0aoqYligg3z8G8MY/edit?usp=sharing Wrapping diagram]<br />
** Insert code into module template (Old VTK code)<br />
** Patch the library that is called (New VTK code)<br />
** Somehow store code in spec and apply when executed?<br />
*** Executed code needs at a minimum access to inputs, outputs, current class<br />
*** Use standard input/output dict and obj reference that the code operates on?<br />
*** Can this be done while keeping the execution abstract?<br />
<br />
* Do new release now?<br />
<br />
=== September 9, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* New release?<br />
** [RR] wants reprounzip<br />
** tej changes & doc<br />
** job monitor & jobmixin fixes<br />
** mongodb<br />
** reprounzip<br />
** warning: MplFigure type is int<br />
** tabledata: add Reshape, DictoToTable & ListToTable, fix ListToTable with numpy arrays<br />
** PythonSource can have same name for an input & output<br />
** don't show spreadsheet at exit<br />
<br />
=== September 2, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Limit autosaves? [https://github.com/VisTrails/VisTrails/pull/1126 #1126]<br />
<br />
=== August 26, 2015 ===<br />
'''Updates'''<br />
* [TE] Added Job support to Parameter Explorations (Requested by Colin), and Mashups<br />
** specify job ids, need to specify different ids for parameter explorations since they have the same version id<br />
** how to deal with parameters passed in on command line<br />
* [General] provenance: should be creating a new version when we execute workflow with changed parameters?<br />
** currently, custom_params annotation stores this in provenance currently<br />
** [RR] API doesn't record provenance if passing in parameters: http://git.io/vsAA5<br />
'''Items to Discuss'''<br />
<br />
=== August 19, 2015 ===<br />
'''Updates'''<br />
* [RR] Improvements to tej, users' guide entry & example [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
* [RR] Internal docs [https://github.com/VisTrails/VisTrails/pull/1116 #1116]<br />
** On ReadTheDocs: [http://vistrails.readthedocs.org/ vistrails.readthedocs.org]<br />
* [TE] Working on Job Monitor documentation<br />
<br />
'''Items to Discuss'''<br />
* [RR] JobMixin and JobMonitor: stable now?<br />
** Definitely needs more tests<br />
* [TE] Stop testing VisTrails 2.0?<br />
<br />
=== August 12, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Internal docs [https://github.com/VisTrails/VisTrails/pull/1116 #1116]<br />
** On ReadTheDocs: [http://vistrails.readthedocs.org/ vistrails.readthedocs.org]<br />
* [RR] JobMixin and JobMonitor: stable now?<br />
** Definitely needs more tests<br />
* [RR] Improvements to tej, users' guide entry & example [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
* [RR] Build broken on Travis; because of IPython 4 released today? ([https://travis-ci.org/VisTrails/VisTrails/builds/75279795 build 992]; [https://github.com/VisTrails/VisTrails/issues/1123 #1123])<br />
<br />
=== August 5, 2015 ===<br />
'''Updates'''<br />
* [TE] Fixes to Jobs<br />
** Could not delete jobs<br />
** Could not run job in group<br />
** Job not reset when calling ModuleSuspended<br />
** Added deleting job from context menu<br />
* [RR] Writing documentation for everything [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
<br />
'''Items to Discuss'''<br />
<br />
=== July 29, 2015 ===<br />
'''Updates'''<br />
* RR still looking into new interpreter thing<br />
** Goal is to take out scheduling logic from Module so it can be split in multiple processes, and so that smarter strategies can be added in time<br />
** This means some work on packages<br />
** Spreadsheet can live in kernel process? Still some UI stuff that will take work (changing configuration, persistent archive's viewer, ...)<br />
* [TE] Problems running examples [https://github.com/VisTrails/VisTrails/pull/1111 #1111]<br />
** Testing of more examples requires additional packages on the test machines.<br />
** Fixed faulty line-ending in PythonSource:s failing on Python 2.6.<br />
** Test suite now testing SUDSWebServices (If web service is down, test suite will fail)<br />
** preferences.py test failed reloading 'dialogs' package, switched to using 'URL'.<br />
<br />
'''Items to Discuss'''<br />
<br />
=== July 22, 2015 ===<br />
'''Updates'''<br />
* [TE] mailing lists back online<br />
* [TE] Working on [https://github.com/VisTrails/VisTrails/issues/1107 #1107]<br />
* [RR] Working on ReproZip package <br />
<br />
'''Items to Discuss'''<br />
* [RR] MongoDB package [https://github.com/VisTrails/VisTrails/pull/1106 #1106]<br />
* [RR] Example for tej docs? [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
<br />
=== July 15, 2015 ===<br />
'''Updates'''<br />
* 2.2.2 released<br />
<br />
'''Items to Discuss'''<br />
* RR is considering executing everything in an IPython kernel (i.e. separate Python interpreter, like the one spawned for a notebook)<br />
** It's a separate process that we can restart without restarting the app/gui<br />
** We can isolate the execution environment (e.g. for the server)<br />
** We can run the whole thing remotely (if your data is elsewhere, just run VisTrails locally and the pipeline elsewhere)<br />
** Pipeline execution no longer makes the interface hang, it just makes the kernel hang (but that's fine)<br />
** We can use notebooks as modules (probably way nicer than the PythonSource module)<br />
** We can run multiple kernels so long as the ports carry things that are serializable<br />
*** meaning we can put the multithreaded-interpreter without all the hacky parts it has now<br />
*** we can run IPython kernels in all the languages IPython supports, [https://github.com/ipython/ipython/wiki/IPython-kernels-for-other-languages currently 46]<br />
<br />
=== July 8, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Buildbot github hook not working after IP address change<br />
* [TE] No reference to VistrailsApplication ([https://github.com/VisTrails/VisTrails/issues/1103 #1103])<br />
** [RR] Added PR [https://github.com/VisTrails/VisTrails/pull/1104 #1104]<br />
* [TE] Reopening VT file after saving with bundled subworkflow won't offer subworkflow upgrade ([https://github.com/VisTrails/VisTrails/issues/1102 #1102])<br />
** allow manual delete to fix right now<br />
** fix this on top of the use-uuid branch<br />
* [TE] Release VisTrails 2.2.1? ([https://github.com/VisTrails/VisTrails/blob/v2.2/CHANGELOG CHANGELOG])<br />
** [https://github.com/VisTrails/VisTrails/pull/1093 output-modules-use-global-dir]<br />
** [https://github.com/VisTrails/VisTrails/milestones/version%202.2 Current issues for 2.2]<br />
<br />
=== July 1, 2015 ===<br />
'''Updates'''<br />
* [TE] PROV fixed<br />
* [TE] Working on subworkflow issues<br />
* [RR] Considering reworking the controller (log vs exception problem, retained upgrades causing interferences) and also the interpreter (IPython?)<br />
<br />
'''Items to Discuss'''<br />
<br />
=== June 24, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* UV-CDAT<br />
* [TE] Can a cyclic workflow be valid? ([https://github.com/VisTrails/VisTrails/issues/1097 #1097])<br />
** focus on disabling the ability to create cyclic pipelines because more things break than just this with a cyclic pipeline<br />
* [RR] Relative paths ([https://github.com/VisTrails/VisTrails/pull/1057 #1057])<br />
** This interacts with the new bundle; how do handle packing files inside the VT bundle?<br />
<br />
=== June 17, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] current_version and reusing existing upgrades are broken ([https://github.com/VisTrails/VisTrails/issues/1095 ticket #1095])<br />
** Could be that export to PROV is using an unflushed upgrade pipeline?<br />
** current_version would then be correct.<br />
** It may work to flush the actions before exporting?<br />
<br />
=== June 10, 2015 ===<br />
'''Updates'''<br />
* DAT: fixed VTK issue on Linux and Mac<br />
* Still crashes on Windows. Need help! Reminder: this works in the VisTrails spreadsheet (QCellPresenter), although no widget get changed there during a drag<br />
** Is it a VTK bug?<br />
** Is it simply impossible to change widgets during the drag, should we do it a different way?<br />
** Did I miss something that is done in VisTrails but somehow not in DAT?<br />
** Low prio, UV-CDAT doesn't run on Windows anyway<br />
<br />
'''Items to Discuss'''<br />
* [TE] Executable Paper ([https://github.com/VisTrails/VisTrails/issues/1086 ticket #1088] [https://github.com/VisTrails/VisTrails/pull/1088 Updated pull request])<br />
* Single instance code and batch mode<br />
<br />
=== June 3, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Executable Paper ([https://github.com/VisTrails/VisTrails/issues/1086 ticket #1088] [https://github.com/VisTrails/VisTrails/pull/1088 pr])<br />
** Requires fixes to command line parameters, Output modules, and batch mode<br />
** How to test this<br />
** Updated missing/outdated flags<br />
** Fixed view issues when generating graphs<br />
* [TE] batch mode<br />
** SpreadsheetOutput not enabled in batch mode. Should we check is_running_gui instead?<br />
** Other instance setting flags from caller<br />
** Is graphsAsPdf replacing spreadsheetDumpPdf?<br />
** graphsAsPdf true by Default?<br />
** Batch mode executing by default (Not needed when generating graphs)<br />
** Re-added workflowInfo as withWorkflowInfo for writing graph and xml workflow<br />
** [DK] batch mode should be outputting to files or stdout, shouldn't always trigger SpreadsheetMode<br />
** execute flag, maybe make execution the default and allow a "--no-execute" if you only want to capture graphs, for example<br />
<br />
=== May 27, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] API question<br />
** http://lists.vistrails.org/pipermail/vistrails-users/2015-May/000357.html<br />
* Status of DAT<br />
** Can add VCS plot to DAT but need to work on configuration windows<br />
** Documentation for graphics templates for UV-CDAT/vcs?<br />
* Qt support?<br />
<br />
=== May 20, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Reviving the DAT, integrating scripting & porting UV-CDAT!<br />
** On [https://github.com/VisTrails/DAT GitHub] ([https://github.com/VisTrails/DAT/issues issues])<br />
** Merging 2 years of development taking longer than expected, but getting there. The plan is to get the patches in VisTrails and never fork again, we never want to get in UV-CDAT's situation (and don't need to).<br />
** VTK cell works fine on Linux but there was flickering on Mac & Windows before; still issues on Mac (Windows status unknown)<br />
** Can get a VCS plot soon (but will need VTK cell fix)<br />
** Integrate in UV-CDAT's build system (so we have cdms, VCS, ...) -> RR can do this, low priority<br />
** How do we integrate scripting?<br />
*** We want to be able to seamlessly make changes to a plot by changing Python code<br />
*** Define new plots by entering Python code without writing modules/packages?<br />
* [RR] UV-CDAT [https://github.com/UV-CDAT/uvcdat/issues/1275 needs] [https://github.com/VisTrails/VisTrails/pull/1073 #1073], please take a look<br />
* [RR] Subworkflow issues<br />
** [https://github.com/VisTrails/VisTrails/issues/1065 #1065]: missing attribute (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/1066 #1066]: groups in subworkflows were ignored when finding dependencies (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/725 #725]: "upgrade subworkflow" button not popping up (fixed); also, namespace issue<br />
** [https://github.com/VisTrails/VisTrails/issues/1071 #1071]: subworkflows don't go through upgrades<br />
* [RR] [https://github.com/VisTrails/VisTrails/issues/1074 #1074]: can't load and edit a single pipeline if upgrade happen<br />
<br />
=== May 13, 2015 ===<br />
'''Updates'''<br />
* [RR] Test skipping whitelist ([https://github.com/VisTrails/VisTrails/pull/1069 #1069]) -- low priority<br />
* [RR] Custom matplotlib modules can't be compatible with both 2.1 and 2.2 ([https://github.com/VisTrails/VisTrails/issues/1067 #1067]); should be fixed for ALPS ([https://github.com/VisTrails/VisTrails/issues/1070 #1070])<br />
<br />
'''Items to Discuss'''<br />
* [RR] UV-CDAT [https://github.com/UV-CDAT/uvcdat/issues/1275 needs] [https://github.com/VisTrails/VisTrails/pull/1073 #1073], please take a look<br />
* [RR] Subworkflow issues<br />
** [https://github.com/VisTrails/VisTrails/issues/1065 #1065]: missing attribute (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/1066 #1066]: groups in subworkflows were ignored when finding dependencies (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/725 #725]: "upgrade subworkflow" button not popping up (fixed); also, namespace issue<br />
** [https://github.com/VisTrails/VisTrails/issues/1071 #1071]: subworkflows don't go through upgrades<br />
* [RR] [https://github.com/VisTrails/VisTrails/issues/1074 #1074]: can't load and edit a single pipeline if upgrade happen<br />
<br />
=== May 7, 2015 ===<br />
'''Updates'''<br />
* [RR] Export/import workflow to Python working!<br />
<br />
'''Items to Discuss'''<br />
* [Claudio] UV-CDAT<br />
** The UV-CDAT project is the biggest user base of VisTrails<br />
** VisTrails package management provides a lot of friction towards people plugging in their code<br />
** Need to make it easy to integrate your random Python scripts in the system without having to deal with all the boilerplate, at least in the first step<br />
** [RR] argues that modules are still good; UV-CDAT shouldn't move towards a purely script-based backend<br />
** [RR] export/import with Python could reduce a lot of that friction by allowing 1) to edit workflow as Python 2) to open up boxes automatically if needed code doesn't match actual modules<br />
** ...<br />
<br />
=== April 29, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] <strike>Abstractions</strike> subworkflows status ([https://github.com/VisTrails/VisTrails/labels/subworkflows tickets])<br />
* [RR] matplotlib compatibility (2.1 & 2.2), [https://github.com/VisTrails/VisTrails/issues/1067 #1067]<br />
** RR to try and fix ALPS matplotlib modules<br />
<br />
=== April 22, 2015 ===<br />
'''Updates'''<br />
* 2.2.0 has been released!<br />
* Windows issue (via email from Ryan)<br />
** issue with manifest file (may be a new file in VTK6?)<br />
** Tommy has regenerated new Windows builds<br />
* Binaries, pypi, and conda released<br />
* [RR] Export as script<br />
* Python sources using VTK need to switch to SetInputData (users should be aware of this)<br />
<br />
=== April 15, 2015 ===<br />
'''Updates'''<br />
* Ready for 2.2.0 (apart from binary/deps issues)<br />
** Missing some libs (scikit-learn, tej, tdparser, SQLAlchemy+connectors<br />
** Windows: runvistrails.py is no longer used, so the PATH is wrong<br />
** Windows: pip is broken, but it probably wouldn't work anyway because of permission issues (disable this?)<br />
* Queries, upgrades and getPipeline() usage ([https://github.com/VisTrails/VisTrails/issues/1054 #1054])<br />
** Getting a pipeline with getPipeline() is not safe: it might return an invalid pipeline<br />
** This is used in many places throughout the code, like queries<br />
** Upgrading would require going through the controller, but that creates new actions<br />
* [TC] Avoid copying a module's output if it's used as input by exactly one downstream module ([https://github.com/VisTrails/VisTrails/issues/1060 #1060]) (useful for big numpy arrays you can update in-place)<br />
<br />
=== April 8, 2015 ===<br />
'''Updates'''<br />
* [DK] Merged RR's changes for output modules (1012 and 1013)<br />
** RR will merge remaining changes, then create v2.2 branch!<br />
<br />
'''Items to Discuss'''<br />
* BNL need numpy array to VTK image<br />
** Looks like VTK has some helpers for this<br />
** We will help if issues arise<br />
** Will contribute back to VisTrails package<br />
* Upgrade issue: [https://github.com/VisTrails/VisTrails/issues/1017 #1017]<br />
** Automatic upgrades should happen between versions of provided upgrades<br />
** Our existing upgrades work around this so it doesn't need to be 2.2.0<br />
* Corner-case VTK modules<br />
** No longer need VTKCell input port, so don't interfere with registry and API anymore<br />
** Still work weirdly, people probably shouldn't use them<br />
** But we have lots of clunky modules since we wrap the whole of VTK; some people might rely on this and know how to use them, let's keep them anyway<br />
** Ready for 2.2.0<br />
<br />
=== April 1, 2015 ===<br />
'''Updates'''<br />
* [RR] UV-CDAT: bugfixing for 2.2, long-term plans: implement scripting import/export in VisTrails, port to UV-CDAT<br />
* Possibly, try to move to regular VisTrails to use new features<br />
<br />
'''Items to Discuss'''<br />
* 2.2 release: nothing much is pending anymore, release next week?<br />
** [TE] vtkExporter classes [https://github.com/VisTrails/VisTrails/issues/1032 #1032]<br />
** [RR] Hiding upgrades in version tree might make it (or might be 2.2.1 so we can test it out) [https://github.com/VisTrails/VisTrails/issues/949 #949]<br />
** Output modules changes to go in<br />
* add note to documentation about order of parameters in VTK<br />
* add issue about exporter upgrades if not already there<br />
<br />
=== March 25, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* 2.2 release<br />
** Checklist on Github: https://github.com/VisTrails/VisTrails/wiki/2.2-Release-Checklist<br />
** Review output modules<br />
** RR has a couple more issues to fix<br />
** Ready to go -- sign app package for OSX? [https://github.com/VisTrails/VisTrails/issues/984 #984] We need access to the Apple Membership team<br />
<br />
=== March 18, 2015 ===<br />
'''Updates'''<br />
* [TE] VTK6 works<br />
<br />
'''Items to Discuss'''<br />
* [RR] Release v2.1.5 with backported tabledata?<br />
** MTA example needs updated tabledata (for JoinTables)<br />
** Google Maps package still not available<br />
* [RR] Work torward v2.2.0?<br />
** changes:<br />
*** new persistence<br />
*** API changes<br />
*** output module changes (upgrades?), maintain cells but try to upgrade<br />
*** not wrapping stuff<br />
*** VTK6? yes<br />
*** JobSubmission stuff?<br />
*** relabeling for upgrades [https://github.com/VisTrails/VisTrails/issues/949 #949]<br />
** makes sense, needs the tree view code to be updated, check selection<br />
** See [https://github.com/VisTrails/VisTrails/wiki/2.2-Release-Checklist 2.2 checklist]<br />
* Discussion of [https://github.com/VisTrails/VisTrails/issues/1016 #1016]<br />
** plumbing between outputs and output modes, how to define a mode that works for many outputs without writing for each output?<br />
<br />
=== March 11, 2015 ===<br />
'''Updates'''<br />
* [RR] persistent_archive done; merge? ([https://github.com/VisTrails/VisTrails/pull/755 #755])<br />
** note about the focus events for widgets<br />
** TE be aware of file_archive for future binaries that include persistent_archive<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/998 New VTK package] finished<br />
<br />
'''Items to Discuss'''<br />
* [RR] Unmark UV-CDAT/VisTrails as a fork of VisTrails/VisTrails? (see last week; decision needed)<br />
** RR email to JF about this<br />
** yes; email sent to Github<br />
* [RR] Switching order of output ports ([https://github.com/VisTrails/VisTrails/pull/1006 #1006])<br />
** added port specs are sorted at a separate spot (Module.*_port_specs properties) than those in the registry (which are sorted in the registry), but those two lists are just combined without respect to sort keys<br />
** need to determine whether the two lists should be merged or remain distinct<br />
** should make sure that order of input ports and output ports makes parallel connections for things with same order<br />
** DK suggests breaking backward compatibility here: workflows still run, can fix easily if a problem in an existing package.<br />
* [RR] Question about output modes ([https://github.com/VisTrails/VisTrails/pull/1007 #1007]), how to integrate in API ([https://github.com/VisTrails/VisTrails/pull/24 #24])<br />
** Should ImageFileMode be removed? ("image" is not a mode, "file" is)<br />
** ImageOutput missing?<br />
** Feel free to change how formats works<br />
* [TE] Test suite segfaults on Fedora 17 virtualbox. Install newer version? (Support ended in 2013)<br />
<br />
=== March 4, 2015 ===<br />
'''Updates'''<br />
* [TE] VTK wrapper<br />
** Works on VTK 5.10<br />
** Still need to test VTK 6<br />
** New general wrappers for python functions and classes <br />
<br />
'''Items to Discuss'''<br />
* [RR] Unmark UV-CDAT/VisTrails as a fork of VisTrails/VisTrails?<br />
** When [http://i.imgur.com/UTY2qfg.png filing a pull request] from UV-CDAT/VisTrails, VisTrails/VisTrails is selected by default<br />
** People keep forgetting to change the default ([https://github.com/VisTrails/VisTrails/pull/956 #956] [https://github.com/VisTrails/VisTrails/pull/968 #968] [https://github.com/VisTrails/VisTrails/pull/999 #999] [https://github.com/VisTrails/VisTrails/pull/1000 #1000] [https://github.com/VisTrails/VisTrails/pull/1003 #1003] [https://github.com/VisTrails/VisTrails/pull/1004 #1004] [https://github.com/VisTrails/VisTrails/pull/1005 #1005])<br />
** Only way to change that is to not have it marked as a fork of VisTrails/VisTrails<br />
** Github staff [https://gist.github.com/remram44/b364c24ef1e462ce6e2f can make that change for us]; should we do it?<br />
** Juliana: comments on this? visibility vs. convenience/annoyance for developers<br />
* [RR] What about [https://github.com/VisTrails/VisTrails/issues/991 #991]?<br />
<br />
=== February 25, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] vtkviewcell for infovis support, can we unify with VTKCell?<br />
** need to test this<br />
* [TE] vtk wrapping<br />
** Mostly finished<br />
** VTK 5.10 produce incorrect results with old wrapping<br />
*** Old wrapper is based mostly on VTK 4<br />
*** Most vtk_examples affected<br />
*** [http://www.vtk.org/Wiki/VTK/VTK_6_Migration/Replacement_of_SetInput]<br />
*** should be able to upgrade from SetInput to SetInputData (need to drop GetOutput and replace with self ports)<br />
*** can we change vtkInstance to just return self and not wrap things<br />
** terminator example not working under 5.8?<br />
* How does VTK wrapping fit into general wrapping framework?<br />
* [RR] new persistence package<br />
<br />
=== February 18, 2015 ===<br />
'''Updates'''<br />
* [RR] New VisTrails API and IPython integration ([https://github.com/VisTrails/VisTrails/pull/24 #24])<br />
<br />
'''Items to Discuss'''<br />
* [TE] VTK wrapping<br />
** [https://gist.github.com/rexissimus/16b74693545b5e7c1035 Benchmarking vtk package]<br />
*** Old: 24.7 seconds <br />
*** New: 10.5 seconds (Except first time that adds 8 sec)<br />
**** The parsing that calls is_abstract (that tries to instanciate all vtk classes) is now only run the first time.<br />
**** get_items_from_sigstring takes 2 seconds, maybe we can use a lookup dict for already computed sigstrings?<br />
** Now using a general python function wrapper<br />
*** VTK classes are wrapped into python function that does not depend on vistrails<br />
*** VTK functions can be executed without vistrails<br />
*** The spec maps functions into vistrails modules, but can also describe wrapping <br />
*** A general python function wrapper that supports <br />
**** kwarg inputs<br />
**** single, list, dict outputs<br />
**** callback for progress reporting<br />
**** temporary file generator for using FilePool<br />
**** optional output generation<br />
*** Creating specs:<br />
***** Create spec by hand<br />
***** Auto-create spec outline (TODO) and manually finish it<br />
***** Dynamically create spec (VTK)<br />
***** Implement documentation wrappers (Can use scikitlearn wrapper to wrap numpydoc) (TODO)<br />
***** Classes as bad functions needs to be wrapped in new functions before they are wrapped. This is different for each package.<br />
****** Classes is hard: Like VTK, and matplotlib. Scikit-learn does still not wrap classes<br />
***** Spec diffing and patching could be done using code from matplotlib.<br />
** Still needs upgrades from old VTK package<br />
*** Is it possible to dynamically wrap functions, e.g, you see a SetFunc and just remove the 'Set' prefix. Or do you need to create a complete mapping?<br />
<br />
=== February 11, 2015 ===<br />
'''Updates'''<br />
* Update from Friday's meeting<br />
** discussed VisTrails internals<br />
** discussed wrapping<br />
*** xml discussion, hard to modify because tied to db code<br />
*** TE has made it possible to add the schema-defined attributes to the intermediate representation<br />
*** higher-level operations on the port specs<br />
** make sure the simple case works<br />
*** [JF] take a simple package with documentation and figure out what the base case for wrapping is<br />
<br />
'''Items to Discuss'''<br />
* [TE] VTK wrapping<br />
** Dynamic loading works<br />
** Reading XML is fast enough, but [https://gist.github.com/rexissimus/8272d3c776ccfa115d82 unserializing data is slow]<br />
** Working on patterns for patching<br />
** Matplotlib has many advanced patterns like argument ordering, nested arguments, alternateSpecs, output types.<br />
*** Having all this in a general wrapper might confuse users?<br />
** [RR] Delay module (except for identifiers) until you need it---e.g. don't deal with port specs, etc. until necessary<br />
* Scripting Support [https://github.com/VisTrails/VisTrails/pull/950 #950]<br />
** [RR] Issue with getting code from modules<br />
** Design a simple solution<br />
** [JF] Couldn't you use modules as black boxes without conversion, just to call into modules/subworkflows easily from e.g. IPython?<br />
*** [RR] This is a job for the API, and a very separate use case. See [https://github.com/VisTrails/VisTrails/pull/24 #24]<br />
<br />
=== February 4, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* Wrapping<br />
** Format to use? Currently XML (like current matplotlib)<br />
*** JSON and YAML have simple "to python dictionary" methods<br />
*** But don't stream<br />
*** YAML a lot easier for humans<br />
** [DK] vtk-new-package also changes parameter names, creates enumerations<br />
*** intermediate schema needs to be extensible<br />
*** packages will want to store there specific infos for compute() method generation<br />
*** also might have specs-altering info, like matplotlib's alternateSpec<br />
** representation to code , registry already has schema for some aspects<br />
** [RR] We might want to see if Module subclasses can be created lazily<br />
*** no need to create all the classes just to register them in the registry and never actually use most of them<br />
*** future effort<br />
* [RR] Where should VisTrails packages live?<br />
** tej installs as 'vistrailspkg.tej', TE installed it as 'userpackages.tej'<br />
** Currently, standard packages are 'vistrails.packages.', user packages are 'userpackages.' and packages loaded through pkg_resources might be anything<br />
** [RR] Use 'vistrailspkg.' everywhere?<br />
** Long-term effort to simplify package distribution/installation (and have VisTrails get them automatically?)<br />
<br />
=== January 28, 2015 ===<br />
'''Updates'''<br />
* T. Caswell to come visit on Fri 6 to discuss wrapping work<br />
<br />
'''Items to Discuss'''<br />
* [TE] New VTK wrapping<br />
** Current code by DK seems a good deal faster<br />
** Generates XML that can be patched/tweaked, generates Python code from it, VisTrails only loads generated Python code<br />
** RR would rather have VisTrails load intermediate representation (= XML) directly, wants to make sure this is not slower<br />
** The goal is to turn the intermediate step into something generic that would be used for every wrapped package (vtk, numpy, matplotlib, sklearn, java) instead of each having its own<br />
** TC has its own code at [https://github.com/Nikea/VTTools/blob/master/vttools/wrap_lib.py github:VTTools] which parses numpy docstrings and generates modules, doesn't yet handle classes or persist anything<br />
* Web crawler<br />
** Right now, TE starts jobs for "start crawler", "stop crawler", "install classifier"<br />
** RR would rather have the crawling be a job as far as VisTrails and tej are concerned<br />
** The whole thing would be one pipeline: load examples, train classifier, start crawler [check for job, kill previous one, upload model, start processes], get snapshot, visualize<br />
*** Need some support in tej and job submission system: long-running jobs, stop a job (wait for it to finish?), restart a job even though results are cached<br />
<br />
=== January 21, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Unified wrapping method discussion [https://github.com/VisTrails/VisTrails/issues/991 #991]<br />
** TE to work on reusable method with intermediate representation, starting with VTK<br />
* [RR] Examples for scikit-learn: JF has an old example using Weka with parameter exploration (not currently in source tree)<br />
** AM's examples are enough<br />
* [AM] scikit-learn package is done, merge it in? [https://github.com/VisTrails/VisTrails/pull/955#issuecomment-70286788 #955]<br />
** RR will merge<br />
* [RR] What should copyright headers say? [https://github.com/VisTrails/VisTrails/pull/994#issuecomment-70173935 #994]<br />
** Let's keep everything in there: Utah/Poly/NYU<br />
<br />
=== January 14, 2015 ===<br />
'''Updates'''<br />
* [TE] Working on classifier<br />
* [RR] Scripting integration, work in progress<br />
<br />
'''Items to Discuss'''<br />
* [RR] Unified wrapping method discussion ([https://github.com/VisTrails/VisTrails/issues/991 #991])<br />
** Let's talk next week, [AM] and [DK] are not here<br />
<br />
=== January 7, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* make sure that we address critical issues, questions, and pending review branches in a timely manner<br />
* scripting support<br />
** [RR] no issues if we want to just keep annotations in the generated code to allow the link back to a workflow<br />
** [RR] can translate from workflow to script, working on script to workflow<br />
** will work for parameter value changes, structural changes require changes to the annotations<br />
** need to publish best practices here<br />
** would be cool to do looping in scripts (easier interface than with workflows)<br />
* notebook support (convert form notebook to workflow)<br />
** RR will sync with FC on this<br />
* Issue with console in built-from-scratch<br />
** [TC] iPython rearranged some of the completion stuff in 2.2 and 2.3<br />
** binary has old version of iPython -> 1.0.0, should we update?<br />
* [TC] automated wrapping of numpy and scipy<br />
** discovered a bunch of malformed documentation in numpy and scipy<br />
** has github repo for vistrails tools<br />
** example modules wrap a bunch of R stuff (not baked in, just how things are)<br />
** will be pushing wrapping logic up<br />
** port names forbidden (window and domain)<br />
** have an import hook to get from yaml directly to VisTrails Modules<br />
** should work for any python modules with well-formed numpy docstrings.<br />
* [Action] should make it clear in documentation that Constant now means serializable not that the value doesn't change (e.g. List)<br />
* [TC] might be interesting to try to build components of matplotlib and accumulate in figure (long-term project, but thinking about how this might work)<br />
* [TE] build and build scripts<br />
** completely automatic, buildbot<br />
** need to set the build machines for the environment we want for the binary<br />
** would virtualenv work here?<br />
** [TC] anaconda can pin versions, potential path to test different configurations<br />
** Q: upload nightly binary builds? A: makes sense, make sure they are well-labeled<br />
* sourceforge stats: e.g. http://sourceforge.net/projects/vistrails/files/vistrails/nightly/vistrails-src-nightly.tar.gz/stats/timeline?dates=2014-01-01+to+2015-01-07<br />
* package issues (see Remi's message)<br />
* [TE] Scope of tej<br />
** Support single ssh commands?<br />
** Queue can be used as a remote machine (crawler is using queue.call*)<br />
* SourceForge stats: http://sourceforge.net/projects/vistrails/files/vistrails/nightly/vistrails-src-nightly.tar.gz/stats/timeline?dates=2014-01-01+to+2015-01-07<br />
<br />
== Older meetings ==<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=11434Development2016-03-23T15:21:49Z<p>Remi: /* March 23, 2016 */ Minutes</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2016 ==<br />
<br />
=== March 23, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Usage reporting almost ready to go<br />
** Need to record whether specific features are used: paramexpore, mashup, query<br />
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation<br />
** How to record size of vistrail?<br />
*** [DK] IdScope already walks over vistrail, use that<br />
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines<br />
<br />
=== March 16, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Upgrades in Query Mode<br />
** Now using code similar to do_version_switch<br />
*** Uses handle_invalid_pipeline<br />
*** Differences: Not using current_pipeline, flushes changes<br />
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)<br />
*** Integrate to avoid repeated code?<br />
*** Use this for query mode?<br />
* [TE] VisTrails 2.2.4 ready for release?<br />
** tej package updated. Others?<br />
* [JF] upgrades and the version tree<br />
* [JF] provenance in Jupyter<br />
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]<br />
<br />
=== March 9, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixing search mode<br />
** String query now matches all by default<br />
*** This caused pipeline queries to never match<br />
** String query can search: User/Notes/Tagname/Time/Module name<br />
*** Should we search module names by default? (Disabled by default because it is expensive)<br />
*** Fixed notes search<br />
** '>' parameter queries in documentation but has been removed<br />
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)<br />
*** Upgrades still a problem<br />
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)<br />
*** [JF] Allow search on past tags<br />
<br />
* [TE] Create new bugfix release<br />
** Finish querying<br />
** Path issue on windows<br />
<br />
* [DK] Look into sqlalchemy and new bundle branches<br />
<br />
=== March 2, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed cache bug with port specs<br />
* [TE] VisTrails Next<br />
** Most packages now work<br />
** No qgis for Python 3<br />
<br />
'''Items to Discuss'''<br />
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== 2015 ==<br />
<br />
=== December 30, 2015 ===<br />
'''Updates'''<br />
* Bokeh package finished<br />
** Can output to Browser/IPython/Spreadsheet (Limited functionality in spreadsheet)<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Bokeh needs QWebEngine (Chromium) in Qt5 which requires PyQt5<br />
*** Many plotting libraries only work in browsers<br />
*** But Chromium may be a big and controversial dependency<br />
*** Or create a browser version of the spreadsheet?<br />
** Some users have requested Python 3<br />
** Not backwards compatible?<br />
** Target for VisTrails 3.0 with new interpreter?<br />
<br />
=== December 16, 2015 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Users list question<br />
* [TE] Wrapping Bokeh<br />
** Added bokeh property parser<br />
*** complements numpydoc parser<br />
*** Not all attributes are available, partially because I am using a pre-release version.<br />
** Wrapped modules<br />
*** bokeh.plotting finished (Backwards like matplotlib: Create plot then draw on it)<br />
*** bokeh.charts mostly done (Missing arguments)<br />
** QWebView mostly works (but no zooming or buttons)<br />
*** It would be better to use external browser or QWebEngineView<br />
** Added improvements to wrapper<br />
*** Generalized port translations (Color/Path to native types)<br />
*** Can use multiple docstring/property parsers simultaneously<br />
<br />
=== December 9, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Looking into wrapper<br />
** Will port TensorFlow<br />
** Doing a pass on the wrapper code & doc<br />
*** UX improvements<br />
** Factorize high-level docstring parsers into core.wrapper? (sphinx, numpy, google)<br />
** Problem with name vs module_name (fails sklearn tests because class_by_name() is broken)<br />
** Looking into compressing specs, lazy registration of modules<br />
<br />
=== December 2, 2015 ===<br />
'''Updates'''<br />
* [TE] Package wrapping<br />
** Wrapped most of numpy/scipy using __all__<br />
** Added class attribute and method access for classes (Can add to class or to separate inspector module)<br />
** TODO: Document wrapping procedure<br />
<br />
'''Items to Discuss'''<br />
<br />
=== November 25, 2015 ===<br />
'''Updates'''<br />
* [RR] New interpreter is coming along<br />
** Everything is a stream<br />
** Task system, eventually work-stealing parallelism<br />
** Streams no longer advance in lock-step<br />
*** Allows for constructions like filter, join, sorted-merge<br />
** depth>1 still to be tested, no current plan to allow this through the module interface<br />
*** Through looping though, can do streams of streams<br />
<br />
'''Items to Discuss'''<br />
* [TE] Package wrapping<br />
** Wrapped most numpy functions (no polynomials)<br />
** and half of scipy classes and functions<br />
** Added functions manually from documentation (no parseable lists?)<br />
** Focusing on data flow constructs (no property/method access for classes)<br />
*** Using spec diff (empty for now) and dynamic parser (stores spec in .vistrails/)<br />
** Python class/function wrapper now stable<br />
** Add google doc parser?<br />
*** Is there a parser available?<br />
<br />
=== November 18, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Package wrapping<br />
** Numpy class wrapping works<br />
*** Basic type is List, since most types are array_likes.<br />
*** TODO: More classes and functions<br />
*** Some operations are in-place and some aren't, and docstring not super clear<br />
*** [RR] It seems only methods mutate, and most have a numpy.xx function equivalent; just manually go through the methods and remove the mutating ones?<br />
** Class wrapper is modular<br />
*** Docstring parser and type string parser can be customized<br />
*** Classes can have optional attribute/method ports<br />
**** Inspectors and attribute/method modules can be created separately<br />
** Use PythonCalc as an example<br />
*** Can be re-implemented as a function with a parseable docstring<br />
*** [RR] PythonCalc is just an example, and it might actually make sense to get rid of it, or do a proper math package with scalar operations (as separate modules, no combobox)<br />
** [RR] TensorFlow to use wrapping as well<br />
*** Very simple (only types 'tensor' & 'variable')<br />
<br />
=== November 11, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Package wrapping<br />
** Wrapping numpy's classes using numpydoc (ndarray)<br />
*** Wrapping constructor arguments, attribute getters/setters as module ports<br />
**** Then how to access attributes afterwards?<br />
***** Use input value port to class modules?<br />
*** Wrap methods as modules? E.g., `ndarray.shape`.<br />
*** Function wrapping is almost a subset of class wrapping<br />
**** May be able to use the same parsing/execution methods<br />
** VTK's non-getter/setter methods could be wrapped as modules<br />
*** We could then remove extra logic in interpreter for keeping function order<br />
** Bokeh uses autogenerated docstrings<br />
*** We may be able to read the specification directly<br />
<br />
* [RR] [https://github.com/VisTrails/VisTrails/pull/1138 TensorFlow package]<br />
** Basic setup working, can execute the Mandelbrot example<br />
** Will autogenerate the operations<br />
<br />
=== November 4, 2015 ===<br />
'''Updates'''<br />
* [RR] Interpreter work requires a fix for [https://github.com/VisTrails/VisTrails/issues/1137 DB issue #1137]<br />
* [DK] Kitware's Resonant [http://www.kitware.com/source/home/post/173]<br />
** [https://github.com/girder/girder Girder]: data management system<br />
** [https://github.com/Kitware/romanesco Romanesco]: execution engine, uses [http://www.celeryproject.org/ Celery] for task management<br />
** Resonant Flow: web app for editing and executing workflows<br />
<br />
'''Items to Discuss'''<br />
* [TE] Added OSX Lion (10.7) VM on build machine using vagrant (seems ok with license?)<br />
** New builds works on Lion<br />
** Change the minimum requirement to 10.7?<br />
* [TE] Package wrapping<br />
* Added upgrade suggestions using 2 spec versions ([http://pastebin.com/30XUU8Gp example])<br />
** using name edit distance to find matches<br />
** Could use something better like port similarity for modules, and type similarity for ports<br />
** Should write python upgrade code?<br />
** Added spec to sklearn package<br />
*** Can now diff spec versions and keep the spec static<br />
*** There will be problems with downgrades and package versions<br />
<br />
=== October 29, 2015 ===<br />
'''Updates'''<br />
* [TE] nightly-build-mac fixed (Needed to approve xcode license)<br />
<br />
'''Items to Discuss'''<br />
* [TE] matplotlib wrapper<br />
** ported to general spec<br />
** Diff can be used on general spec<br />
*** Only needed minor changes<br />
*** Fixed indexing bug that corrupted specs<br />
*** Can now move diff tools to core/wrapper<br />
** Add new plots?<br />
*** New plots in mpl 1.3: eventplot<br />
*** New plots in mpl 1.4: angle_spectrum, magnitude_spectrum, phase_spectrum', violinplot<br />
*** No new plots in mpl 1.5 (from looking at boilerplate.py)<br />
<br />
=== October 21, 2015 ===<br />
'''Updates'''<br />
* [RR] Alexis has arrived, will be working with [RR] on a more efficient interpreter<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1133 New matplotlib package]<br />
** Supporting multiple package versions<br />
*** Check which version can be loaded [https://github.com/VisTrails/VisTrails/pull/1135 #1135]<br />
*** We cannot show version requirements in the list of packages because old packages are loaded when importing the codepath<br />
** Visual diff does not work well, but we can use the spec differ to see differences.<br />
** Unifying vtk and matplotlib wrapping specs (WIP)<br />
*** And porting matplotlibs spec differ to work on the general spec<br />
*** Spec differ can then be used on vtk and other packages in the future <br />
*** Will enable automatic upgrade generation<br />
* [RR] Rework interpreter<br />
** restore abstraction between interpreter/module code<br />
** build looping, streaming, caching into the interpreter<br />
** rework cache<br />
** look into parallelism once basic functionality is there<br />
<br />
=== October 14, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1133 New matplotlib package] (adding numpydoc parser)<br />
<br />
=== October 7, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://github.com/VisTrails/VisTrails/tree/fix-matplotlib-parser New matplotlib parser] (adding numpydoc parser)<br />
** Added numpydoc attribute parser for plots.<br />
*** TODO: Need better port spec reconciliation with call signature parser.<br />
*** We can create a general numpydoc parser, but attribute types are unique to matplotlib.<br />
** Needs package versioning<br />
*** New spec will not support matplotlib < v1.4 due to changed path to axes classes.<br />
*** Load spec version corresponding to installed matplotlib version?<br />
*** Need version downgrades?<br />
<br />
=== September 30, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Updating Matplotlib parser<br />
** Matplotlib docstring parser fails on numpy docstrings<br />
** I have [https://github.com/VisTrails/VisTrails/commit/eae8b0c16260810587e47c41cab53552cbb3cf0a added basic numpydoc parsing]<br />
*** Only used by a few docstrings so far<br />
*** Parsing uses many sources (class tables, signatures, ACCEPTS, method docstrings, definition parsing).<br />
** Caswell said they were thinking about moving to traitlets, but this is not ready yet<br />
<br />
=== September 23, 2015 ===<br />
'''Updates'''<br />
* [TE] VisTrails 2.2.3 released<br />
** Also have pushed to PyPI, binstar, etc.<br />
** Sourceforge vs. GitHub<br />
*** should be able to host releases on GitHub<br />
*** nightly binaries pushed to sf each night<br />
*** old binaries?<br />
* [TE] Re-wrapping MatPlotlib<br />
** Keep static generation - Docstrings are brittle beween matplotlib versions<br />
** Keep generating the executable classes - but create functions, not vistrails Modules <br />
*** Why is there so much patching?<br />
** Re-implemented Alternate PortSpec for InputPortSpec<br />
*** It will now inherit specs from parent (No need to reimplement)<br />
** Module upgrades?<br />
<br />
'''Items to Discuss'''<br />
<br />
=== September 16, 2015 ===<br />
'''Updates'''<br />
* [TE] Job Monitor tests and documentation done<br />
** OK to leave document package in packages directory?<br />
<br />
'''Items to Discuss'''<br />
* [TE] Library wrapping: How to do code patching? [https://docs.google.com/drawings/d/1Sh3eOXot6v2y880jHKQrqXqAwr-0aoqYligg3z8G8MY/edit?usp=sharing Wrapping diagram]<br />
** Insert code into module template (Old VTK code)<br />
** Patch the library that is called (New VTK code)<br />
** Somehow store code in spec and apply when executed?<br />
*** Executed code needs at a minimum access to inputs, outputs, current class<br />
*** Use standard input/output dict and obj reference that the code operates on?<br />
*** Can this be done while keeping the execution abstract?<br />
<br />
* Do new release now?<br />
<br />
=== September 9, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* New release?<br />
** [RR] wants reprounzip<br />
** tej changes & doc<br />
** job monitor & jobmixin fixes<br />
** mongodb<br />
** reprounzip<br />
** warning: MplFigure type is int<br />
** tabledata: add Reshape, DictoToTable & ListToTable, fix ListToTable with numpy arrays<br />
** PythonSource can have same name for an input & output<br />
** don't show spreadsheet at exit<br />
<br />
=== September 2, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Limit autosaves? [https://github.com/VisTrails/VisTrails/pull/1126 #1126]<br />
<br />
=== August 26, 2015 ===<br />
'''Updates'''<br />
* [TE] Added Job support to Parameter Explorations (Requested by Colin), and Mashups<br />
** specify job ids, need to specify different ids for parameter explorations since they have the same version id<br />
** how to deal with parameters passed in on command line<br />
* [General] provenance: should be creating a new version when we execute workflow with changed parameters?<br />
** currently, custom_params annotation stores this in provenance currently<br />
** [RR] API doesn't record provenance if passing in parameters: http://git.io/vsAA5<br />
'''Items to Discuss'''<br />
<br />
=== August 19, 2015 ===<br />
'''Updates'''<br />
* [RR] Improvements to tej, users' guide entry & example [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
* [RR] Internal docs [https://github.com/VisTrails/VisTrails/pull/1116 #1116]<br />
** On ReadTheDocs: [http://vistrails.readthedocs.org/ vistrails.readthedocs.org]<br />
* [TE] Working on Job Monitor documentation<br />
<br />
'''Items to Discuss'''<br />
* [RR] JobMixin and JobMonitor: stable now?<br />
** Definitely needs more tests<br />
* [TE] Stop testing VisTrails 2.0?<br />
<br />
=== August 12, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Internal docs [https://github.com/VisTrails/VisTrails/pull/1116 #1116]<br />
** On ReadTheDocs: [http://vistrails.readthedocs.org/ vistrails.readthedocs.org]<br />
* [RR] JobMixin and JobMonitor: stable now?<br />
** Definitely needs more tests<br />
* [RR] Improvements to tej, users' guide entry & example [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
* [RR] Build broken on Travis; because of IPython 4 released today? ([https://travis-ci.org/VisTrails/VisTrails/builds/75279795 build 992]; [https://github.com/VisTrails/VisTrails/issues/1123 #1123])<br />
<br />
=== August 5, 2015 ===<br />
'''Updates'''<br />
* [TE] Fixes to Jobs<br />
** Could not delete jobs<br />
** Could not run job in group<br />
** Job not reset when calling ModuleSuspended<br />
** Added deleting job from context menu<br />
* [RR] Writing documentation for everything [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
<br />
'''Items to Discuss'''<br />
<br />
=== July 29, 2015 ===<br />
'''Updates'''<br />
* RR still looking into new interpreter thing<br />
** Goal is to take out scheduling logic from Module so it can be split in multiple processes, and so that smarter strategies can be added in time<br />
** This means some work on packages<br />
** Spreadsheet can live in kernel process? Still some UI stuff that will take work (changing configuration, persistent archive's viewer, ...)<br />
* [TE] Problems running examples [https://github.com/VisTrails/VisTrails/pull/1111 #1111]<br />
** Testing of more examples requires additional packages on the test machines.<br />
** Fixed faulty line-ending in PythonSource:s failing on Python 2.6.<br />
** Test suite now testing SUDSWebServices (If web service is down, test suite will fail)<br />
** preferences.py test failed reloading 'dialogs' package, switched to using 'URL'.<br />
<br />
'''Items to Discuss'''<br />
<br />
=== July 22, 2015 ===<br />
'''Updates'''<br />
* [TE] mailing lists back online<br />
* [TE] Working on [https://github.com/VisTrails/VisTrails/issues/1107 #1107]<br />
* [RR] Working on ReproZip package <br />
<br />
'''Items to Discuss'''<br />
* [RR] MongoDB package [https://github.com/VisTrails/VisTrails/pull/1106 #1106]<br />
* [RR] Example for tej docs? [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
<br />
=== July 15, 2015 ===<br />
'''Updates'''<br />
* 2.2.2 released<br />
<br />
'''Items to Discuss'''<br />
* RR is considering executing everything in an IPython kernel (i.e. separate Python interpreter, like the one spawned for a notebook)<br />
** It's a separate process that we can restart without restarting the app/gui<br />
** We can isolate the execution environment (e.g. for the server)<br />
** We can run the whole thing remotely (if your data is elsewhere, just run VisTrails locally and the pipeline elsewhere)<br />
** Pipeline execution no longer makes the interface hang, it just makes the kernel hang (but that's fine)<br />
** We can use notebooks as modules (probably way nicer than the PythonSource module)<br />
** We can run multiple kernels so long as the ports carry things that are serializable<br />
*** meaning we can put the multithreaded-interpreter without all the hacky parts it has now<br />
*** we can run IPython kernels in all the languages IPython supports, [https://github.com/ipython/ipython/wiki/IPython-kernels-for-other-languages currently 46]<br />
<br />
=== July 8, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Buildbot github hook not working after IP address change<br />
* [TE] No reference to VistrailsApplication ([https://github.com/VisTrails/VisTrails/issues/1103 #1103])<br />
** [RR] Added PR [https://github.com/VisTrails/VisTrails/pull/1104 #1104]<br />
* [TE] Reopening VT file after saving with bundled subworkflow won't offer subworkflow upgrade ([https://github.com/VisTrails/VisTrails/issues/1102 #1102])<br />
** allow manual delete to fix right now<br />
** fix this on top of the use-uuid branch<br />
* [TE] Release VisTrails 2.2.1? ([https://github.com/VisTrails/VisTrails/blob/v2.2/CHANGELOG CHANGELOG])<br />
** [https://github.com/VisTrails/VisTrails/pull/1093 output-modules-use-global-dir]<br />
** [https://github.com/VisTrails/VisTrails/milestones/version%202.2 Current issues for 2.2]<br />
<br />
=== July 1, 2015 ===<br />
'''Updates'''<br />
* [TE] PROV fixed<br />
* [TE] Working on subworkflow issues<br />
* [RR] Considering reworking the controller (log vs exception problem, retained upgrades causing interferences) and also the interpreter (IPython?)<br />
<br />
'''Items to Discuss'''<br />
<br />
=== June 24, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* UV-CDAT<br />
* [TE] Can a cyclic workflow be valid? ([https://github.com/VisTrails/VisTrails/issues/1097 #1097])<br />
** focus on disabling the ability to create cyclic pipelines because more things break than just this with a cyclic pipeline<br />
* [RR] Relative paths ([https://github.com/VisTrails/VisTrails/pull/1057 #1057])<br />
** This interacts with the new bundle; how do handle packing files inside the VT bundle?<br />
<br />
=== June 17, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] current_version and reusing existing upgrades are broken ([https://github.com/VisTrails/VisTrails/issues/1095 ticket #1095])<br />
** Could be that export to PROV is using an unflushed upgrade pipeline?<br />
** current_version would then be correct.<br />
** It may work to flush the actions before exporting?<br />
<br />
=== June 10, 2015 ===<br />
'''Updates'''<br />
* DAT: fixed VTK issue on Linux and Mac<br />
* Still crashes on Windows. Need help! Reminder: this works in the VisTrails spreadsheet (QCellPresenter), although no widget get changed there during a drag<br />
** Is it a VTK bug?<br />
** Is it simply impossible to change widgets during the drag, should we do it a different way?<br />
** Did I miss something that is done in VisTrails but somehow not in DAT?<br />
** Low prio, UV-CDAT doesn't run on Windows anyway<br />
<br />
'''Items to Discuss'''<br />
* [TE] Executable Paper ([https://github.com/VisTrails/VisTrails/issues/1086 ticket #1088] [https://github.com/VisTrails/VisTrails/pull/1088 Updated pull request])<br />
* Single instance code and batch mode<br />
<br />
=== June 3, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Executable Paper ([https://github.com/VisTrails/VisTrails/issues/1086 ticket #1088] [https://github.com/VisTrails/VisTrails/pull/1088 pr])<br />
** Requires fixes to command line parameters, Output modules, and batch mode<br />
** How to test this<br />
** Updated missing/outdated flags<br />
** Fixed view issues when generating graphs<br />
* [TE] batch mode<br />
** SpreadsheetOutput not enabled in batch mode. Should we check is_running_gui instead?<br />
** Other instance setting flags from caller<br />
** Is graphsAsPdf replacing spreadsheetDumpPdf?<br />
** graphsAsPdf true by Default?<br />
** Batch mode executing by default (Not needed when generating graphs)<br />
** Re-added workflowInfo as withWorkflowInfo for writing graph and xml workflow<br />
** [DK] batch mode should be outputting to files or stdout, shouldn't always trigger SpreadsheetMode<br />
** execute flag, maybe make execution the default and allow a "--no-execute" if you only want to capture graphs, for example<br />
<br />
=== May 27, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] API question<br />
** http://lists.vistrails.org/pipermail/vistrails-users/2015-May/000357.html<br />
* Status of DAT<br />
** Can add VCS plot to DAT but need to work on configuration windows<br />
** Documentation for graphics templates for UV-CDAT/vcs?<br />
* Qt support?<br />
<br />
=== May 20, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Reviving the DAT, integrating scripting & porting UV-CDAT!<br />
** On [https://github.com/VisTrails/DAT GitHub] ([https://github.com/VisTrails/DAT/issues issues])<br />
** Merging 2 years of development taking longer than expected, but getting there. The plan is to get the patches in VisTrails and never fork again, we never want to get in UV-CDAT's situation (and don't need to).<br />
** VTK cell works fine on Linux but there was flickering on Mac & Windows before; still issues on Mac (Windows status unknown)<br />
** Can get a VCS plot soon (but will need VTK cell fix)<br />
** Integrate in UV-CDAT's build system (so we have cdms, VCS, ...) -> RR can do this, low priority<br />
** How do we integrate scripting?<br />
*** We want to be able to seamlessly make changes to a plot by changing Python code<br />
*** Define new plots by entering Python code without writing modules/packages?<br />
* [RR] UV-CDAT [https://github.com/UV-CDAT/uvcdat/issues/1275 needs] [https://github.com/VisTrails/VisTrails/pull/1073 #1073], please take a look<br />
* [RR] Subworkflow issues<br />
** [https://github.com/VisTrails/VisTrails/issues/1065 #1065]: missing attribute (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/1066 #1066]: groups in subworkflows were ignored when finding dependencies (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/725 #725]: "upgrade subworkflow" button not popping up (fixed); also, namespace issue<br />
** [https://github.com/VisTrails/VisTrails/issues/1071 #1071]: subworkflows don't go through upgrades<br />
* [RR] [https://github.com/VisTrails/VisTrails/issues/1074 #1074]: can't load and edit a single pipeline if upgrade happen<br />
<br />
=== May 13, 2015 ===<br />
'''Updates'''<br />
* [RR] Test skipping whitelist ([https://github.com/VisTrails/VisTrails/pull/1069 #1069]) -- low priority<br />
* [RR] Custom matplotlib modules can't be compatible with both 2.1 and 2.2 ([https://github.com/VisTrails/VisTrails/issues/1067 #1067]); should be fixed for ALPS ([https://github.com/VisTrails/VisTrails/issues/1070 #1070])<br />
<br />
'''Items to Discuss'''<br />
* [RR] UV-CDAT [https://github.com/UV-CDAT/uvcdat/issues/1275 needs] [https://github.com/VisTrails/VisTrails/pull/1073 #1073], please take a look<br />
* [RR] Subworkflow issues<br />
** [https://github.com/VisTrails/VisTrails/issues/1065 #1065]: missing attribute (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/1066 #1066]: groups in subworkflows were ignored when finding dependencies (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/725 #725]: "upgrade subworkflow" button not popping up (fixed); also, namespace issue<br />
** [https://github.com/VisTrails/VisTrails/issues/1071 #1071]: subworkflows don't go through upgrades<br />
* [RR] [https://github.com/VisTrails/VisTrails/issues/1074 #1074]: can't load and edit a single pipeline if upgrade happen<br />
<br />
=== May 7, 2015 ===<br />
'''Updates'''<br />
* [RR] Export/import workflow to Python working!<br />
<br />
'''Items to Discuss'''<br />
* [Claudio] UV-CDAT<br />
** The UV-CDAT project is the biggest user base of VisTrails<br />
** VisTrails package management provides a lot of friction towards people plugging in their code<br />
** Need to make it easy to integrate your random Python scripts in the system without having to deal with all the boilerplate, at least in the first step<br />
** [RR] argues that modules are still good; UV-CDAT shouldn't move towards a purely script-based backend<br />
** [RR] export/import with Python could reduce a lot of that friction by allowing 1) to edit workflow as Python 2) to open up boxes automatically if needed code doesn't match actual modules<br />
** ...<br />
<br />
=== April 29, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] <strike>Abstractions</strike> subworkflows status ([https://github.com/VisTrails/VisTrails/labels/subworkflows tickets])<br />
* [RR] matplotlib compatibility (2.1 & 2.2), [https://github.com/VisTrails/VisTrails/issues/1067 #1067]<br />
** RR to try and fix ALPS matplotlib modules<br />
<br />
=== April 22, 2015 ===<br />
'''Updates'''<br />
* 2.2.0 has been released!<br />
* Windows issue (via email from Ryan)<br />
** issue with manifest file (may be a new file in VTK6?)<br />
** Tommy has regenerated new Windows builds<br />
* Binaries, pypi, and conda released<br />
* [RR] Export as script<br />
* Python sources using VTK need to switch to SetInputData (users should be aware of this)<br />
<br />
=== April 15, 2015 ===<br />
'''Updates'''<br />
* Ready for 2.2.0 (apart from binary/deps issues)<br />
** Missing some libs (scikit-learn, tej, tdparser, SQLAlchemy+connectors<br />
** Windows: runvistrails.py is no longer used, so the PATH is wrong<br />
** Windows: pip is broken, but it probably wouldn't work anyway because of permission issues (disable this?)<br />
* Queries, upgrades and getPipeline() usage ([https://github.com/VisTrails/VisTrails/issues/1054 #1054])<br />
** Getting a pipeline with getPipeline() is not safe: it might return an invalid pipeline<br />
** This is used in many places throughout the code, like queries<br />
** Upgrading would require going through the controller, but that creates new actions<br />
* [TC] Avoid copying a module's output if it's used as input by exactly one downstream module ([https://github.com/VisTrails/VisTrails/issues/1060 #1060]) (useful for big numpy arrays you can update in-place)<br />
<br />
=== April 8, 2015 ===<br />
'''Updates'''<br />
* [DK] Merged RR's changes for output modules (1012 and 1013)<br />
** RR will merge remaining changes, then create v2.2 branch!<br />
<br />
'''Items to Discuss'''<br />
* BNL need numpy array to VTK image<br />
** Looks like VTK has some helpers for this<br />
** We will help if issues arise<br />
** Will contribute back to VisTrails package<br />
* Upgrade issue: [https://github.com/VisTrails/VisTrails/issues/1017 #1017]<br />
** Automatic upgrades should happen between versions of provided upgrades<br />
** Our existing upgrades work around this so it doesn't need to be 2.2.0<br />
* Corner-case VTK modules<br />
** No longer need VTKCell input port, so don't interfere with registry and API anymore<br />
** Still work weirdly, people probably shouldn't use them<br />
** But we have lots of clunky modules since we wrap the whole of VTK; some people might rely on this and know how to use them, let's keep them anyway<br />
** Ready for 2.2.0<br />
<br />
=== April 1, 2015 ===<br />
'''Updates'''<br />
* [RR] UV-CDAT: bugfixing for 2.2, long-term plans: implement scripting import/export in VisTrails, port to UV-CDAT<br />
* Possibly, try to move to regular VisTrails to use new features<br />
<br />
'''Items to Discuss'''<br />
* 2.2 release: nothing much is pending anymore, release next week?<br />
** [TE] vtkExporter classes [https://github.com/VisTrails/VisTrails/issues/1032 #1032]<br />
** [RR] Hiding upgrades in version tree might make it (or might be 2.2.1 so we can test it out) [https://github.com/VisTrails/VisTrails/issues/949 #949]<br />
** Output modules changes to go in<br />
* add note to documentation about order of parameters in VTK<br />
* add issue about exporter upgrades if not already there<br />
<br />
=== March 25, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* 2.2 release<br />
** Checklist on Github: https://github.com/VisTrails/VisTrails/wiki/2.2-Release-Checklist<br />
** Review output modules<br />
** RR has a couple more issues to fix<br />
** Ready to go -- sign app package for OSX? [https://github.com/VisTrails/VisTrails/issues/984 #984] We need access to the Apple Membership team<br />
<br />
=== March 18, 2015 ===<br />
'''Updates'''<br />
* [TE] VTK6 works<br />
<br />
'''Items to Discuss'''<br />
* [RR] Release v2.1.5 with backported tabledata?<br />
** MTA example needs updated tabledata (for JoinTables)<br />
** Google Maps package still not available<br />
* [RR] Work torward v2.2.0?<br />
** changes:<br />
*** new persistence<br />
*** API changes<br />
*** output module changes (upgrades?), maintain cells but try to upgrade<br />
*** not wrapping stuff<br />
*** VTK6? yes<br />
*** JobSubmission stuff?<br />
*** relabeling for upgrades [https://github.com/VisTrails/VisTrails/issues/949 #949]<br />
** makes sense, needs the tree view code to be updated, check selection<br />
** See [https://github.com/VisTrails/VisTrails/wiki/2.2-Release-Checklist 2.2 checklist]<br />
* Discussion of [https://github.com/VisTrails/VisTrails/issues/1016 #1016]<br />
** plumbing between outputs and output modes, how to define a mode that works for many outputs without writing for each output?<br />
<br />
=== March 11, 2015 ===<br />
'''Updates'''<br />
* [RR] persistent_archive done; merge? ([https://github.com/VisTrails/VisTrails/pull/755 #755])<br />
** note about the focus events for widgets<br />
** TE be aware of file_archive for future binaries that include persistent_archive<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/998 New VTK package] finished<br />
<br />
'''Items to Discuss'''<br />
* [RR] Unmark UV-CDAT/VisTrails as a fork of VisTrails/VisTrails? (see last week; decision needed)<br />
** RR email to JF about this<br />
** yes; email sent to Github<br />
* [RR] Switching order of output ports ([https://github.com/VisTrails/VisTrails/pull/1006 #1006])<br />
** added port specs are sorted at a separate spot (Module.*_port_specs properties) than those in the registry (which are sorted in the registry), but those two lists are just combined without respect to sort keys<br />
** need to determine whether the two lists should be merged or remain distinct<br />
** should make sure that order of input ports and output ports makes parallel connections for things with same order<br />
** DK suggests breaking backward compatibility here: workflows still run, can fix easily if a problem in an existing package.<br />
* [RR] Question about output modes ([https://github.com/VisTrails/VisTrails/pull/1007 #1007]), how to integrate in API ([https://github.com/VisTrails/VisTrails/pull/24 #24])<br />
** Should ImageFileMode be removed? ("image" is not a mode, "file" is)<br />
** ImageOutput missing?<br />
** Feel free to change how formats works<br />
* [TE] Test suite segfaults on Fedora 17 virtualbox. Install newer version? (Support ended in 2013)<br />
<br />
=== March 4, 2015 ===<br />
'''Updates'''<br />
* [TE] VTK wrapper<br />
** Works on VTK 5.10<br />
** Still need to test VTK 6<br />
** New general wrappers for python functions and classes <br />
<br />
'''Items to Discuss'''<br />
* [RR] Unmark UV-CDAT/VisTrails as a fork of VisTrails/VisTrails?<br />
** When [http://i.imgur.com/UTY2qfg.png filing a pull request] from UV-CDAT/VisTrails, VisTrails/VisTrails is selected by default<br />
** People keep forgetting to change the default ([https://github.com/VisTrails/VisTrails/pull/956 #956] [https://github.com/VisTrails/VisTrails/pull/968 #968] [https://github.com/VisTrails/VisTrails/pull/999 #999] [https://github.com/VisTrails/VisTrails/pull/1000 #1000] [https://github.com/VisTrails/VisTrails/pull/1003 #1003] [https://github.com/VisTrails/VisTrails/pull/1004 #1004] [https://github.com/VisTrails/VisTrails/pull/1005 #1005])<br />
** Only way to change that is to not have it marked as a fork of VisTrails/VisTrails<br />
** Github staff [https://gist.github.com/remram44/b364c24ef1e462ce6e2f can make that change for us]; should we do it?<br />
** Juliana: comments on this? visibility vs. convenience/annoyance for developers<br />
* [RR] What about [https://github.com/VisTrails/VisTrails/issues/991 #991]?<br />
<br />
=== February 25, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] vtkviewcell for infovis support, can we unify with VTKCell?<br />
** need to test this<br />
* [TE] vtk wrapping<br />
** Mostly finished<br />
** VTK 5.10 produce incorrect results with old wrapping<br />
*** Old wrapper is based mostly on VTK 4<br />
*** Most vtk_examples affected<br />
*** [http://www.vtk.org/Wiki/VTK/VTK_6_Migration/Replacement_of_SetInput]<br />
*** should be able to upgrade from SetInput to SetInputData (need to drop GetOutput and replace with self ports)<br />
*** can we change vtkInstance to just return self and not wrap things<br />
** terminator example not working under 5.8?<br />
* How does VTK wrapping fit into general wrapping framework?<br />
* [RR] new persistence package<br />
<br />
=== February 18, 2015 ===<br />
'''Updates'''<br />
* [RR] New VisTrails API and IPython integration ([https://github.com/VisTrails/VisTrails/pull/24 #24])<br />
<br />
'''Items to Discuss'''<br />
* [TE] VTK wrapping<br />
** [https://gist.github.com/rexissimus/16b74693545b5e7c1035 Benchmarking vtk package]<br />
*** Old: 24.7 seconds <br />
*** New: 10.5 seconds (Except first time that adds 8 sec)<br />
**** The parsing that calls is_abstract (that tries to instanciate all vtk classes) is now only run the first time.<br />
**** get_items_from_sigstring takes 2 seconds, maybe we can use a lookup dict for already computed sigstrings?<br />
** Now using a general python function wrapper<br />
*** VTK classes are wrapped into python function that does not depend on vistrails<br />
*** VTK functions can be executed without vistrails<br />
*** The spec maps functions into vistrails modules, but can also describe wrapping <br />
*** A general python function wrapper that supports <br />
**** kwarg inputs<br />
**** single, list, dict outputs<br />
**** callback for progress reporting<br />
**** temporary file generator for using FilePool<br />
**** optional output generation<br />
*** Creating specs:<br />
***** Create spec by hand<br />
***** Auto-create spec outline (TODO) and manually finish it<br />
***** Dynamically create spec (VTK)<br />
***** Implement documentation wrappers (Can use scikitlearn wrapper to wrap numpydoc) (TODO)<br />
***** Classes as bad functions needs to be wrapped in new functions before they are wrapped. This is different for each package.<br />
****** Classes is hard: Like VTK, and matplotlib. Scikit-learn does still not wrap classes<br />
***** Spec diffing and patching could be done using code from matplotlib.<br />
** Still needs upgrades from old VTK package<br />
*** Is it possible to dynamically wrap functions, e.g, you see a SetFunc and just remove the 'Set' prefix. Or do you need to create a complete mapping?<br />
<br />
=== February 11, 2015 ===<br />
'''Updates'''<br />
* Update from Friday's meeting<br />
** discussed VisTrails internals<br />
** discussed wrapping<br />
*** xml discussion, hard to modify because tied to db code<br />
*** TE has made it possible to add the schema-defined attributes to the intermediate representation<br />
*** higher-level operations on the port specs<br />
** make sure the simple case works<br />
*** [JF] take a simple package with documentation and figure out what the base case for wrapping is<br />
<br />
'''Items to Discuss'''<br />
* [TE] VTK wrapping<br />
** Dynamic loading works<br />
** Reading XML is fast enough, but [https://gist.github.com/rexissimus/8272d3c776ccfa115d82 unserializing data is slow]<br />
** Working on patterns for patching<br />
** Matplotlib has many advanced patterns like argument ordering, nested arguments, alternateSpecs, output types.<br />
*** Having all this in a general wrapper might confuse users?<br />
** [RR] Delay module (except for identifiers) until you need it---e.g. don't deal with port specs, etc. until necessary<br />
* Scripting Support [https://github.com/VisTrails/VisTrails/pull/950 #950]<br />
** [RR] Issue with getting code from modules<br />
** Design a simple solution<br />
** [JF] Couldn't you use modules as black boxes without conversion, just to call into modules/subworkflows easily from e.g. IPython?<br />
*** [RR] This is a job for the API, and a very separate use case. See [https://github.com/VisTrails/VisTrails/pull/24 #24]<br />
<br />
=== February 4, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* Wrapping<br />
** Format to use? Currently XML (like current matplotlib)<br />
*** JSON and YAML have simple "to python dictionary" methods<br />
*** But don't stream<br />
*** YAML a lot easier for humans<br />
** [DK] vtk-new-package also changes parameter names, creates enumerations<br />
*** intermediate schema needs to be extensible<br />
*** packages will want to store there specific infos for compute() method generation<br />
*** also might have specs-altering info, like matplotlib's alternateSpec<br />
** representation to code , registry already has schema for some aspects<br />
** [RR] We might want to see if Module subclasses can be created lazily<br />
*** no need to create all the classes just to register them in the registry and never actually use most of them<br />
*** future effort<br />
* [RR] Where should VisTrails packages live?<br />
** tej installs as 'vistrailspkg.tej', TE installed it as 'userpackages.tej'<br />
** Currently, standard packages are 'vistrails.packages.', user packages are 'userpackages.' and packages loaded through pkg_resources might be anything<br />
** [RR] Use 'vistrailspkg.' everywhere?<br />
** Long-term effort to simplify package distribution/installation (and have VisTrails get them automatically?)<br />
<br />
=== January 28, 2015 ===<br />
'''Updates'''<br />
* T. Caswell to come visit on Fri 6 to discuss wrapping work<br />
<br />
'''Items to Discuss'''<br />
* [TE] New VTK wrapping<br />
** Current code by DK seems a good deal faster<br />
** Generates XML that can be patched/tweaked, generates Python code from it, VisTrails only loads generated Python code<br />
** RR would rather have VisTrails load intermediate representation (= XML) directly, wants to make sure this is not slower<br />
** The goal is to turn the intermediate step into something generic that would be used for every wrapped package (vtk, numpy, matplotlib, sklearn, java) instead of each having its own<br />
** TC has its own code at [https://github.com/Nikea/VTTools/blob/master/vttools/wrap_lib.py github:VTTools] which parses numpy docstrings and generates modules, doesn't yet handle classes or persist anything<br />
* Web crawler<br />
** Right now, TE starts jobs for "start crawler", "stop crawler", "install classifier"<br />
** RR would rather have the crawling be a job as far as VisTrails and tej are concerned<br />
** The whole thing would be one pipeline: load examples, train classifier, start crawler [check for job, kill previous one, upload model, start processes], get snapshot, visualize<br />
*** Need some support in tej and job submission system: long-running jobs, stop a job (wait for it to finish?), restart a job even though results are cached<br />
<br />
=== January 21, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Unified wrapping method discussion [https://github.com/VisTrails/VisTrails/issues/991 #991]<br />
** TE to work on reusable method with intermediate representation, starting with VTK<br />
* [RR] Examples for scikit-learn: JF has an old example using Weka with parameter exploration (not currently in source tree)<br />
** AM's examples are enough<br />
* [AM] scikit-learn package is done, merge it in? [https://github.com/VisTrails/VisTrails/pull/955#issuecomment-70286788 #955]<br />
** RR will merge<br />
* [RR] What should copyright headers say? [https://github.com/VisTrails/VisTrails/pull/994#issuecomment-70173935 #994]<br />
** Let's keep everything in there: Utah/Poly/NYU<br />
<br />
=== January 14, 2015 ===<br />
'''Updates'''<br />
* [TE] Working on classifier<br />
* [RR] Scripting integration, work in progress<br />
<br />
'''Items to Discuss'''<br />
* [RR] Unified wrapping method discussion ([https://github.com/VisTrails/VisTrails/issues/991 #991])<br />
** Let's talk next week, [AM] and [DK] are not here<br />
<br />
=== January 7, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* make sure that we address critical issues, questions, and pending review branches in a timely manner<br />
* scripting support<br />
** [RR] no issues if we want to just keep annotations in the generated code to allow the link back to a workflow<br />
** [RR] can translate from workflow to script, working on script to workflow<br />
** will work for parameter value changes, structural changes require changes to the annotations<br />
** need to publish best practices here<br />
** would be cool to do looping in scripts (easier interface than with workflows)<br />
* notebook support (convert form notebook to workflow)<br />
** RR will sync with FC on this<br />
* Issue with console in built-from-scratch<br />
** [TC] iPython rearranged some of the completion stuff in 2.2 and 2.3<br />
** binary has old version of iPython -> 1.0.0, should we update?<br />
* [TC] automated wrapping of numpy and scipy<br />
** discovered a bunch of malformed documentation in numpy and scipy<br />
** has github repo for vistrails tools<br />
** example modules wrap a bunch of R stuff (not baked in, just how things are)<br />
** will be pushing wrapping logic up<br />
** port names forbidden (window and domain)<br />
** have an import hook to get from yaml directly to VisTrails Modules<br />
** should work for any python modules with well-formed numpy docstrings.<br />
* [Action] should make it clear in documentation that Constant now means serializable not that the value doesn't change (e.g. List)<br />
* [TC] might be interesting to try to build components of matplotlib and accumulate in figure (long-term project, but thinking about how this might work)<br />
* [TE] build and build scripts<br />
** completely automatic, buildbot<br />
** need to set the build machines for the environment we want for the binary<br />
** would virtualenv work here?<br />
** [TC] anaconda can pin versions, potential path to test different configurations<br />
** Q: upload nightly binary builds? A: makes sense, make sure they are well-labeled<br />
* sourceforge stats: e.g. http://sourceforge.net/projects/vistrails/files/vistrails/nightly/vistrails-src-nightly.tar.gz/stats/timeline?dates=2014-01-01+to+2015-01-07<br />
* package issues (see Remi's message)<br />
* [TE] Scope of tej<br />
** Support single ssh commands?<br />
** Queue can be used as a remote machine (crawler is using queue.call*)<br />
* SourceForge stats: http://sourceforge.net/projects/vistrails/files/vistrails/nightly/vistrails-src-nightly.tar.gz/stats/timeline?dates=2014-01-01+to+2015-01-07<br />
<br />
== Older meetings ==<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remihttps://www.vistrails.org//index.php?title=Development&diff=11297Development2016-02-23T19:26:47Z<p>Remi: /* February 23, 2016 */</p>
<hr />
<div><div class="nonumtoc toclimit-1"><br />
__TOC__<br />
</div><br />
<br />
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]<br />
<br />
== 2016 ==<br />
<br />
=== February 23, 2016 ===<br />
'''Updates'''<br />
* [TE] Fixed PythonSource 100% CPU bug<br />
* [TE] VisTrails Next<br />
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples<br />
*** Now uses REST service, tabledata, and Google Maps<br />
*** Web Service replaced with DownloadFile<br />
<br />
'''Items to Discuss'''<br />
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.<br />
** New 2.2 release soon?<br />
* [RR] Anonymous usage reports<br />
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]<br />
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?<br />
* noWorkflow - provenance for python scripts<br />
** Recent work on tracking dependencies<br />
* Also integrates ReproZip with Jupyter<br />
** [http://mybinder.org mybinder]<br />
*** Use with ReproZip to automatically build Jupyter notebook interfaces<br />
* Daves comments on Jupyter integration<br />
** It is possible to identify cell order on the server side<br />
** Not trivial to track order of cells on client side<br />
* How does Galaxy workflows relate to VisTrails?<br />
** Packaging in Galaxy is hard<br />
** Galaxy for biomedical workflows, difficulty making it general<br />
* Action items<br />
** How difficult to track cell dependencies in Jupyter notebooks?<br />
** Use noWorkflow to track cell dependencies?<br />
** How difficult to modify mybinder to use ReproZip packages?<br />
<br />
=== February 17, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Fixed Group memory usage<br />
** Modules in groups are supposed to be freed when done executing<br />
*** Prevented by logger that stored references to the modules (fixed, now using id(module))<br />
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)<br />
* [TE] VisTrails Next<br />
** Removed deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** PythonSources will be upgraded<br />
*** Fixes for Python < 3<br />
*** fixes for VTK < 6<br />
** TODOs<br />
*** Remove RemoteQ<br />
**** Port Hadoop to tej (Already have pbs)<br />
*** Merge package wrapper<br />
*** Finish SQLAlchemy branch<br />
*** Finish new bundle branch<br />
** [JF] Will contact Jeff and Matthias about moving to Python 3<br />
* [JF] Query by Example is broken<br />
* [JF] VisTrails Data Cleaning Project <br />
** Use VisTrails to process and explore task-specific data while keeping provenance.<br />
** Remi's Idea<br />
*** Use VisTrails as the backend in a Jupyter notebook<br />
*** Overcome problems with executing cells in random order<br />
**** Interesting research topic, but little interest has yet been noted<br />
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)<br />
*** Impose restrictions to notebook cell boundaries<br />
*** It will only work for functional operations<br />
*** Can we fix, or detect, non-functional (mutating) operations?<br />
*** Analyse code and data flow and see if it mutates data in unexpected ways?<br />
*** Mutable operations can be created in many ways in Python<br />
* Next meeting Tuesday 11:30<br />
<br />
=== February 10, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* VisTrails Next<br />
** Test suite passes<br />
*** Python 3 less deterministic, module upgrade order random.<br />
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...<br />
** All of our packages should run fine :)<br />
** VTK thumbnail comparison works<br />
*** Rendering bug on some systems<br />
**** QVTKCell now inherits QGLWidget<br />
*** Old thumbnails had wrong height due to spreadsheetwindow being too small<br />
**** Fixed by regenerating thumbnails<br />
*** Silenced deprecation messages when generating modules<br />
*** Many new classes in VTK 7<br />
** Remove deprecated code<br />
*** Core imports that should be gui imports<br />
*** GetInputFromPort etc.<br />
**** Will break existing PythonSources<br />
**** Try to upgrade?<br />
*** Old-style packages<br />
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)<br />
***** Read info.py without importing package?<br />
*** Fixes for Python < 3<br />
*** Fixes for Qt < 5<br />
*** fixes for VTK < 7<br />
*** Remove self port?<br />
<br />
=== February 3, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]<br />
<br />
* VisTrails Next<br />
** All tests but thumbnail comparison passes<br />
** Still issues with gui and more packages needs to be tested<br />
** PythonSource and MplSource upgraded with lib2to3<br />
** Keep contrib packages? Update?<br />
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]<br />
*** Supports Python 3<br />
*** Will hopefully get into conda and others soon<br />
*** Deprecates a bunch of modules<br />
<br />
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next<br />
** Runs!<br />
** Uses conda<br />
*** Many levels of requirements<br />
**** apt packages<br />
**** conda packages<br />
**** python packages in requirements.txt<br />
**** python packages in setup.py<br />
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!<br />
*** But useful if we want to use conda in our own build process.<br />
** Lots of errors I do not get locally?<br />
<br />
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]<br />
** Have you used any alternatives to vistrails?<br />
*** Relevant? More options?<br />
** How often do you use any of these particular features in VisTrails?<br />
*** Add/Remove alternatives?<br />
** How important would you rank the following potential improvements to VisTrails?<br />
*** Add alternatives?<br />
<br />
=== January 27, 2016 ===<br />
'''Updates'''<br />
* New interpreter<br />
** Working prototype soon<br />
** Looping not completely done (and port combination mode: pairwise/cartesian...)<br />
** Need to integrate with VisTrails<br />
*** Compatibility with existing modules<br />
*** Use log controller<br />
*** Want to cache at multiple levels (whole stream/single element of stream)<br />
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)<br />
<br />
'''Items to Discuss'''<br />
* User Survey<br />
<br />
* ZMQ failing on windows (vistrails-users)<br />
<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]<br />
** Testsuite almost passes<br />
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.<br />
** Fixed issues<br />
*** Signatures are now bytes (convert to string to show in widget)<br />
*** Cannot compare different types, e.g., (None < 1)<br />
**** PortSpec.depth and min/maxConn could be None. (Backport!)<br />
**** cell.row could be None<br />
*** Lots of unicode/bytes fixes to serializations<br />
**** We have encryption implementations in the code! (keychain)<br />
*** Lots of small fixes to package method differences<br />
** Current issues<br />
*** logging widget empty?<br />
*** version tree signals messed up<br />
*** provenance view legend icons black<br />
*** Text do not fit in some widgets<br />
*** VTK thumbnail comparison fails<br />
**** VTK Cells have some faces in wrong direction?<br />
**** Because running in VM?<br />
** Installing on Ubuntu LTS/Latest Mint<br />
*** Install latest matplotlib from pip<br />
*** Install VTK 7.0.0rc2 from source<br />
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)<br />
<br />
=== January 20, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Backporting features from Python 3 + PyQt5 + Qt5<br />
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])<br />
*** One unicode test failing<br />
*** Problem saving to xml<br />
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])<br />
*** Updates print to functions 'from __future__ import print_function'<br />
*** Updates exception calls to use 'as'<br />
*** removes most usages of 'long' (mostly used unnecessarily)<br />
**** Only needed for 32-bit support<br />
*** Fixes relative imports (from __future__ import absolute_import)<br />
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])<br />
*** Updated all signals to new-style<br />
* If we go 2 to 3, this could break packages<br />
** means we should schedule any package format changes for the same time<br />
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])<br />
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)<br />
** putting package identifier as xml?<br />
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)<br />
** version the APIs?<br />
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?<br />
* file formats on 2 to 3:<br />
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2<br />
<br />
=== January 13, 2016 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]<br />
** Basic GUI and execution works<br />
** Minimal work to support 2to3<br />
** Hard to support both PyQt4 and PyQt5<br />
<br />
=== January 6, 2016 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]<br />
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.<br />
**** Next version of VTK 6 is rumored to support Python 3<br />
**** There is a development branch of VTK 6 supporting Python 3<br />
*** Can travis build this?<br />
**** We can skip tests for WebEngine and VTK if needed<br />
** Python 3 [https://python3wos.appspot.com/ showstoppers]<br />
*** No MySQL-python (sqlalchemy supports mysql through other drivers)<br />
*** No suds (there are active forks but we could also drop support?)<br />
* Options for python2/3 support:<br />
** Interpreter could be python2/python3 based on packages<br />
** six layer works but not the easiest to maintain (e.g. in reprozip)<br />
* Drop support for win-64<br />
* Update copyright headers to 2016<br />
<br />
== 2015 ==<br />
<br />
=== December 30, 2015 ===<br />
'''Updates'''<br />
* Bokeh package finished<br />
** Can output to Browser/IPython/Spreadsheet (Limited functionality in spreadsheet)<br />
'''Items to Discuss'''<br />
* Updating VisTrails dependencies<br />
** Bokeh needs QWebEngine (Chromium) in Qt5 which requires PyQt5<br />
*** Many plotting libraries only work in browsers<br />
*** But Chromium may be a big and controversial dependency<br />
*** Or create a browser version of the spreadsheet?<br />
** Some users have requested Python 3<br />
** Not backwards compatible?<br />
** Target for VisTrails 3.0 with new interpreter?<br />
<br />
=== December 16, 2015 ===<br />
'''Updates'''<br />
'''Items to Discuss'''<br />
* Users list question<br />
* [TE] Wrapping Bokeh<br />
** Added bokeh property parser<br />
*** complements numpydoc parser<br />
*** Not all attributes are available, partially because I am using a pre-release version.<br />
** Wrapped modules<br />
*** bokeh.plotting finished (Backwards like matplotlib: Create plot then draw on it)<br />
*** bokeh.charts mostly done (Missing arguments)<br />
** QWebView mostly works (but no zooming or buttons)<br />
*** It would be better to use external browser or QWebEngineView<br />
** Added improvements to wrapper<br />
*** Generalized port translations (Color/Path to native types)<br />
*** Can use multiple docstring/property parsers simultaneously<br />
<br />
=== December 9, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Looking into wrapper<br />
** Will port TensorFlow<br />
** Doing a pass on the wrapper code & doc<br />
*** UX improvements<br />
** Factorize high-level docstring parsers into core.wrapper? (sphinx, numpy, google)<br />
** Problem with name vs module_name (fails sklearn tests because class_by_name() is broken)<br />
** Looking into compressing specs, lazy registration of modules<br />
<br />
=== December 2, 2015 ===<br />
'''Updates'''<br />
* [TE] Package wrapping<br />
** Wrapped most of numpy/scipy using __all__<br />
** Added class attribute and method access for classes (Can add to class or to separate inspector module)<br />
** TODO: Document wrapping procedure<br />
<br />
'''Items to Discuss'''<br />
<br />
=== November 25, 2015 ===<br />
'''Updates'''<br />
* [RR] New interpreter is coming along<br />
** Everything is a stream<br />
** Task system, eventually work-stealing parallelism<br />
** Streams no longer advance in lock-step<br />
*** Allows for constructions like filter, join, sorted-merge<br />
** depth>1 still to be tested, no current plan to allow this through the module interface<br />
*** Through looping though, can do streams of streams<br />
<br />
'''Items to Discuss'''<br />
* [TE] Package wrapping<br />
** Wrapped most numpy functions (no polynomials)<br />
** and half of scipy classes and functions<br />
** Added functions manually from documentation (no parseable lists?)<br />
** Focusing on data flow constructs (no property/method access for classes)<br />
*** Using spec diff (empty for now) and dynamic parser (stores spec in .vistrails/)<br />
** Python class/function wrapper now stable<br />
** Add google doc parser?<br />
*** Is there a parser available?<br />
<br />
=== November 18, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Package wrapping<br />
** Numpy class wrapping works<br />
*** Basic type is List, since most types are array_likes.<br />
*** TODO: More classes and functions<br />
*** Some operations are in-place and some aren't, and docstring not super clear<br />
*** [RR] It seems only methods mutate, and most have a numpy.xx function equivalent; just manually go through the methods and remove the mutating ones?<br />
** Class wrapper is modular<br />
*** Docstring parser and type string parser can be customized<br />
*** Classes can have optional attribute/method ports<br />
**** Inspectors and attribute/method modules can be created separately<br />
** Use PythonCalc as an example<br />
*** Can be re-implemented as a function with a parseable docstring<br />
*** [RR] PythonCalc is just an example, and it might actually make sense to get rid of it, or do a proper math package with scalar operations (as separate modules, no combobox)<br />
** [RR] TensorFlow to use wrapping as well<br />
*** Very simple (only types 'tensor' & 'variable')<br />
<br />
=== November 11, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Package wrapping<br />
** Wrapping numpy's classes using numpydoc (ndarray)<br />
*** Wrapping constructor arguments, attribute getters/setters as module ports<br />
**** Then how to access attributes afterwards?<br />
***** Use input value port to class modules?<br />
*** Wrap methods as modules? E.g., `ndarray.shape`.<br />
*** Function wrapping is almost a subset of class wrapping<br />
**** May be able to use the same parsing/execution methods<br />
** VTK's non-getter/setter methods could be wrapped as modules<br />
*** We could then remove extra logic in interpreter for keeping function order<br />
** Bokeh uses autogenerated docstrings<br />
*** We may be able to read the specification directly<br />
<br />
* [RR] [https://github.com/VisTrails/VisTrails/pull/1138 TensorFlow package]<br />
** Basic setup working, can execute the Mandelbrot example<br />
** Will autogenerate the operations<br />
<br />
=== November 4, 2015 ===<br />
'''Updates'''<br />
* [RR] Interpreter work requires a fix for [https://github.com/VisTrails/VisTrails/issues/1137 DB issue #1137]<br />
* [DK] Kitware's Resonant [http://www.kitware.com/source/home/post/173]<br />
** [https://github.com/girder/girder Girder]: data management system<br />
** [https://github.com/Kitware/romanesco Romanesco]: execution engine, uses [http://www.celeryproject.org/ Celery] for task management<br />
** Resonant Flow: web app for editing and executing workflows<br />
<br />
'''Items to Discuss'''<br />
* [TE] Added OSX Lion (10.7) VM on build machine using vagrant (seems ok with license?)<br />
** New builds works on Lion<br />
** Change the minimum requirement to 10.7?<br />
* [TE] Package wrapping<br />
* Added upgrade suggestions using 2 spec versions ([http://pastebin.com/30XUU8Gp example])<br />
** using name edit distance to find matches<br />
** Could use something better like port similarity for modules, and type similarity for ports<br />
** Should write python upgrade code?<br />
** Added spec to sklearn package<br />
*** Can now diff spec versions and keep the spec static<br />
*** There will be problems with downgrades and package versions<br />
<br />
=== October 29, 2015 ===<br />
'''Updates'''<br />
* [TE] nightly-build-mac fixed (Needed to approve xcode license)<br />
<br />
'''Items to Discuss'''<br />
* [TE] matplotlib wrapper<br />
** ported to general spec<br />
** Diff can be used on general spec<br />
*** Only needed minor changes<br />
*** Fixed indexing bug that corrupted specs<br />
*** Can now move diff tools to core/wrapper<br />
** Add new plots?<br />
*** New plots in mpl 1.3: eventplot<br />
*** New plots in mpl 1.4: angle_spectrum, magnitude_spectrum, phase_spectrum', violinplot<br />
*** No new plots in mpl 1.5 (from looking at boilerplate.py)<br />
<br />
=== October 21, 2015 ===<br />
'''Updates'''<br />
* [RR] Alexis has arrived, will be working with [RR] on a more efficient interpreter<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1133 New matplotlib package]<br />
** Supporting multiple package versions<br />
*** Check which version can be loaded [https://github.com/VisTrails/VisTrails/pull/1135 #1135]<br />
*** We cannot show version requirements in the list of packages because old packages are loaded when importing the codepath<br />
** Visual diff does not work well, but we can use the spec differ to see differences.<br />
** Unifying vtk and matplotlib wrapping specs (WIP)<br />
*** And porting matplotlibs spec differ to work on the general spec<br />
*** Spec differ can then be used on vtk and other packages in the future <br />
*** Will enable automatic upgrade generation<br />
* [RR] Rework interpreter<br />
** restore abstraction between interpreter/module code<br />
** build looping, streaming, caching into the interpreter<br />
** rework cache<br />
** look into parallelism once basic functionality is there<br />
<br />
=== October 14, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/1133 New matplotlib package] (adding numpydoc parser)<br />
<br />
=== October 7, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] [https://github.com/VisTrails/VisTrails/tree/fix-matplotlib-parser New matplotlib parser] (adding numpydoc parser)<br />
** Added numpydoc attribute parser for plots.<br />
*** TODO: Need better port spec reconciliation with call signature parser.<br />
*** We can create a general numpydoc parser, but attribute types are unique to matplotlib.<br />
** Needs package versioning<br />
*** New spec will not support matplotlib < v1.4 due to changed path to axes classes.<br />
*** Load spec version corresponding to installed matplotlib version?<br />
*** Need version downgrades?<br />
<br />
=== September 30, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Updating Matplotlib parser<br />
** Matplotlib docstring parser fails on numpy docstrings<br />
** I have [https://github.com/VisTrails/VisTrails/commit/eae8b0c16260810587e47c41cab53552cbb3cf0a added basic numpydoc parsing]<br />
*** Only used by a few docstrings so far<br />
*** Parsing uses many sources (class tables, signatures, ACCEPTS, method docstrings, definition parsing).<br />
** Caswell said they were thinking about moving to traitlets, but this is not ready yet<br />
<br />
=== September 23, 2015 ===<br />
'''Updates'''<br />
* [TE] VisTrails 2.2.3 released<br />
** Also have pushed to PyPI, binstar, etc.<br />
** Sourceforge vs. GitHub<br />
*** should be able to host releases on GitHub<br />
*** nightly binaries pushed to sf each night<br />
*** old binaries?<br />
* [TE] Re-wrapping MatPlotlib<br />
** Keep static generation - Docstrings are brittle beween matplotlib versions<br />
** Keep generating the executable classes - but create functions, not vistrails Modules <br />
*** Why is there so much patching?<br />
** Re-implemented Alternate PortSpec for InputPortSpec<br />
*** It will now inherit specs from parent (No need to reimplement)<br />
** Module upgrades?<br />
<br />
'''Items to Discuss'''<br />
<br />
=== September 16, 2015 ===<br />
'''Updates'''<br />
* [TE] Job Monitor tests and documentation done<br />
** OK to leave document package in packages directory?<br />
<br />
'''Items to Discuss'''<br />
* [TE] Library wrapping: How to do code patching? [https://docs.google.com/drawings/d/1Sh3eOXot6v2y880jHKQrqXqAwr-0aoqYligg3z8G8MY/edit?usp=sharing Wrapping diagram]<br />
** Insert code into module template (Old VTK code)<br />
** Patch the library that is called (New VTK code)<br />
** Somehow store code in spec and apply when executed?<br />
*** Executed code needs at a minimum access to inputs, outputs, current class<br />
*** Use standard input/output dict and obj reference that the code operates on?<br />
*** Can this be done while keeping the execution abstract?<br />
<br />
* Do new release now?<br />
<br />
=== September 9, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* New release?<br />
** [RR] wants reprounzip<br />
** tej changes & doc<br />
** job monitor & jobmixin fixes<br />
** mongodb<br />
** reprounzip<br />
** warning: MplFigure type is int<br />
** tabledata: add Reshape, DictoToTable & ListToTable, fix ListToTable with numpy arrays<br />
** PythonSource can have same name for an input & output<br />
** don't show spreadsheet at exit<br />
<br />
=== September 2, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Limit autosaves? [https://github.com/VisTrails/VisTrails/pull/1126 #1126]<br />
<br />
=== August 26, 2015 ===<br />
'''Updates'''<br />
* [TE] Added Job support to Parameter Explorations (Requested by Colin), and Mashups<br />
** specify job ids, need to specify different ids for parameter explorations since they have the same version id<br />
** how to deal with parameters passed in on command line<br />
* [General] provenance: should be creating a new version when we execute workflow with changed parameters?<br />
** currently, custom_params annotation stores this in provenance currently<br />
** [RR] API doesn't record provenance if passing in parameters: http://git.io/vsAA5<br />
'''Items to Discuss'''<br />
<br />
=== August 19, 2015 ===<br />
'''Updates'''<br />
* [RR] Improvements to tej, users' guide entry & example [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
* [RR] Internal docs [https://github.com/VisTrails/VisTrails/pull/1116 #1116]<br />
** On ReadTheDocs: [http://vistrails.readthedocs.org/ vistrails.readthedocs.org]<br />
* [TE] Working on Job Monitor documentation<br />
<br />
'''Items to Discuss'''<br />
* [RR] JobMixin and JobMonitor: stable now?<br />
** Definitely needs more tests<br />
* [TE] Stop testing VisTrails 2.0?<br />
<br />
=== August 12, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Internal docs [https://github.com/VisTrails/VisTrails/pull/1116 #1116]<br />
** On ReadTheDocs: [http://vistrails.readthedocs.org/ vistrails.readthedocs.org]<br />
* [RR] JobMixin and JobMonitor: stable now?<br />
** Definitely needs more tests<br />
* [RR] Improvements to tej, users' guide entry & example [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
* [RR] Build broken on Travis; because of IPython 4 released today? ([https://travis-ci.org/VisTrails/VisTrails/builds/75279795 build 992]; [https://github.com/VisTrails/VisTrails/issues/1123 #1123])<br />
<br />
=== August 5, 2015 ===<br />
'''Updates'''<br />
* [TE] Fixes to Jobs<br />
** Could not delete jobs<br />
** Could not run job in group<br />
** Job not reset when calling ModuleSuspended<br />
** Added deleting job from context menu<br />
* [RR] Writing documentation for everything [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
<br />
'''Items to Discuss'''<br />
<br />
=== July 29, 2015 ===<br />
'''Updates'''<br />
* RR still looking into new interpreter thing<br />
** Goal is to take out scheduling logic from Module so it can be split in multiple processes, and so that smarter strategies can be added in time<br />
** This means some work on packages<br />
** Spreadsheet can live in kernel process? Still some UI stuff that will take work (changing configuration, persistent archive's viewer, ...)<br />
* [TE] Problems running examples [https://github.com/VisTrails/VisTrails/pull/1111 #1111]<br />
** Testing of more examples requires additional packages on the test machines.<br />
** Fixed faulty line-ending in PythonSource:s failing on Python 2.6.<br />
** Test suite now testing SUDSWebServices (If web service is down, test suite will fail)<br />
** preferences.py test failed reloading 'dialogs' package, switched to using 'URL'.<br />
<br />
'''Items to Discuss'''<br />
<br />
=== July 22, 2015 ===<br />
'''Updates'''<br />
* [TE] mailing lists back online<br />
* [TE] Working on [https://github.com/VisTrails/VisTrails/issues/1107 #1107]<br />
* [RR] Working on ReproZip package <br />
<br />
'''Items to Discuss'''<br />
* [RR] MongoDB package [https://github.com/VisTrails/VisTrails/pull/1106 #1106]<br />
* [RR] Example for tej docs? [https://github.com/VisTrails/VisTrails/pull/1105 #1105]<br />
<br />
=== July 15, 2015 ===<br />
'''Updates'''<br />
* 2.2.2 released<br />
<br />
'''Items to Discuss'''<br />
* RR is considering executing everything in an IPython kernel (i.e. separate Python interpreter, like the one spawned for a notebook)<br />
** It's a separate process that we can restart without restarting the app/gui<br />
** We can isolate the execution environment (e.g. for the server)<br />
** We can run the whole thing remotely (if your data is elsewhere, just run VisTrails locally and the pipeline elsewhere)<br />
** Pipeline execution no longer makes the interface hang, it just makes the kernel hang (but that's fine)<br />
** We can use notebooks as modules (probably way nicer than the PythonSource module)<br />
** We can run multiple kernels so long as the ports carry things that are serializable<br />
*** meaning we can put the multithreaded-interpreter without all the hacky parts it has now<br />
*** we can run IPython kernels in all the languages IPython supports, [https://github.com/ipython/ipython/wiki/IPython-kernels-for-other-languages currently 46]<br />
<br />
=== July 8, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Buildbot github hook not working after IP address change<br />
* [TE] No reference to VistrailsApplication ([https://github.com/VisTrails/VisTrails/issues/1103 #1103])<br />
** [RR] Added PR [https://github.com/VisTrails/VisTrails/pull/1104 #1104]<br />
* [TE] Reopening VT file after saving with bundled subworkflow won't offer subworkflow upgrade ([https://github.com/VisTrails/VisTrails/issues/1102 #1102])<br />
** allow manual delete to fix right now<br />
** fix this on top of the use-uuid branch<br />
* [TE] Release VisTrails 2.2.1? ([https://github.com/VisTrails/VisTrails/blob/v2.2/CHANGELOG CHANGELOG])<br />
** [https://github.com/VisTrails/VisTrails/pull/1093 output-modules-use-global-dir]<br />
** [https://github.com/VisTrails/VisTrails/milestones/version%202.2 Current issues for 2.2]<br />
<br />
=== July 1, 2015 ===<br />
'''Updates'''<br />
* [TE] PROV fixed<br />
* [TE] Working on subworkflow issues<br />
* [RR] Considering reworking the controller (log vs exception problem, retained upgrades causing interferences) and also the interpreter (IPython?)<br />
<br />
'''Items to Discuss'''<br />
<br />
=== June 24, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* UV-CDAT<br />
* [TE] Can a cyclic workflow be valid? ([https://github.com/VisTrails/VisTrails/issues/1097 #1097])<br />
** focus on disabling the ability to create cyclic pipelines because more things break than just this with a cyclic pipeline<br />
* [RR] Relative paths ([https://github.com/VisTrails/VisTrails/pull/1057 #1057])<br />
** This interacts with the new bundle; how do handle packing files inside the VT bundle?<br />
<br />
=== June 17, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] current_version and reusing existing upgrades are broken ([https://github.com/VisTrails/VisTrails/issues/1095 ticket #1095])<br />
** Could be that export to PROV is using an unflushed upgrade pipeline?<br />
** current_version would then be correct.<br />
** It may work to flush the actions before exporting?<br />
<br />
=== June 10, 2015 ===<br />
'''Updates'''<br />
* DAT: fixed VTK issue on Linux and Mac<br />
* Still crashes on Windows. Need help! Reminder: this works in the VisTrails spreadsheet (QCellPresenter), although no widget get changed there during a drag<br />
** Is it a VTK bug?<br />
** Is it simply impossible to change widgets during the drag, should we do it a different way?<br />
** Did I miss something that is done in VisTrails but somehow not in DAT?<br />
** Low prio, UV-CDAT doesn't run on Windows anyway<br />
<br />
'''Items to Discuss'''<br />
* [TE] Executable Paper ([https://github.com/VisTrails/VisTrails/issues/1086 ticket #1088] [https://github.com/VisTrails/VisTrails/pull/1088 Updated pull request])<br />
* Single instance code and batch mode<br />
<br />
=== June 3, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] Executable Paper ([https://github.com/VisTrails/VisTrails/issues/1086 ticket #1088] [https://github.com/VisTrails/VisTrails/pull/1088 pr])<br />
** Requires fixes to command line parameters, Output modules, and batch mode<br />
** How to test this<br />
** Updated missing/outdated flags<br />
** Fixed view issues when generating graphs<br />
* [TE] batch mode<br />
** SpreadsheetOutput not enabled in batch mode. Should we check is_running_gui instead?<br />
** Other instance setting flags from caller<br />
** Is graphsAsPdf replacing spreadsheetDumpPdf?<br />
** graphsAsPdf true by Default?<br />
** Batch mode executing by default (Not needed when generating graphs)<br />
** Re-added workflowInfo as withWorkflowInfo for writing graph and xml workflow<br />
** [DK] batch mode should be outputting to files or stdout, shouldn't always trigger SpreadsheetMode<br />
** execute flag, maybe make execution the default and allow a "--no-execute" if you only want to capture graphs, for example<br />
<br />
=== May 27, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [vistrails-users] API question<br />
** http://lists.vistrails.org/pipermail/vistrails-users/2015-May/000357.html<br />
* Status of DAT<br />
** Can add VCS plot to DAT but need to work on configuration windows<br />
** Documentation for graphics templates for UV-CDAT/vcs?<br />
* Qt support?<br />
<br />
=== May 20, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Reviving the DAT, integrating scripting & porting UV-CDAT!<br />
** On [https://github.com/VisTrails/DAT GitHub] ([https://github.com/VisTrails/DAT/issues issues])<br />
** Merging 2 years of development taking longer than expected, but getting there. The plan is to get the patches in VisTrails and never fork again, we never want to get in UV-CDAT's situation (and don't need to).<br />
** VTK cell works fine on Linux but there was flickering on Mac & Windows before; still issues on Mac (Windows status unknown)<br />
** Can get a VCS plot soon (but will need VTK cell fix)<br />
** Integrate in UV-CDAT's build system (so we have cdms, VCS, ...) -> RR can do this, low priority<br />
** How do we integrate scripting?<br />
*** We want to be able to seamlessly make changes to a plot by changing Python code<br />
*** Define new plots by entering Python code without writing modules/packages?<br />
* [RR] UV-CDAT [https://github.com/UV-CDAT/uvcdat/issues/1275 needs] [https://github.com/VisTrails/VisTrails/pull/1073 #1073], please take a look<br />
* [RR] Subworkflow issues<br />
** [https://github.com/VisTrails/VisTrails/issues/1065 #1065]: missing attribute (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/1066 #1066]: groups in subworkflows were ignored when finding dependencies (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/725 #725]: "upgrade subworkflow" button not popping up (fixed); also, namespace issue<br />
** [https://github.com/VisTrails/VisTrails/issues/1071 #1071]: subworkflows don't go through upgrades<br />
* [RR] [https://github.com/VisTrails/VisTrails/issues/1074 #1074]: can't load and edit a single pipeline if upgrade happen<br />
<br />
=== May 13, 2015 ===<br />
'''Updates'''<br />
* [RR] Test skipping whitelist ([https://github.com/VisTrails/VisTrails/pull/1069 #1069]) -- low priority<br />
* [RR] Custom matplotlib modules can't be compatible with both 2.1 and 2.2 ([https://github.com/VisTrails/VisTrails/issues/1067 #1067]); should be fixed for ALPS ([https://github.com/VisTrails/VisTrails/issues/1070 #1070])<br />
<br />
'''Items to Discuss'''<br />
* [RR] UV-CDAT [https://github.com/UV-CDAT/uvcdat/issues/1275 needs] [https://github.com/VisTrails/VisTrails/pull/1073 #1073], please take a look<br />
* [RR] Subworkflow issues<br />
** [https://github.com/VisTrails/VisTrails/issues/1065 #1065]: missing attribute (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/1066 #1066]: groups in subworkflows were ignored when finding dependencies (fixed)<br />
** [https://github.com/VisTrails/VisTrails/issues/725 #725]: "upgrade subworkflow" button not popping up (fixed); also, namespace issue<br />
** [https://github.com/VisTrails/VisTrails/issues/1071 #1071]: subworkflows don't go through upgrades<br />
* [RR] [https://github.com/VisTrails/VisTrails/issues/1074 #1074]: can't load and edit a single pipeline if upgrade happen<br />
<br />
=== May 7, 2015 ===<br />
'''Updates'''<br />
* [RR] Export/import workflow to Python working!<br />
<br />
'''Items to Discuss'''<br />
* [Claudio] UV-CDAT<br />
** The UV-CDAT project is the biggest user base of VisTrails<br />
** VisTrails package management provides a lot of friction towards people plugging in their code<br />
** Need to make it easy to integrate your random Python scripts in the system without having to deal with all the boilerplate, at least in the first step<br />
** [RR] argues that modules are still good; UV-CDAT shouldn't move towards a purely script-based backend<br />
** [RR] export/import with Python could reduce a lot of that friction by allowing 1) to edit workflow as Python 2) to open up boxes automatically if needed code doesn't match actual modules<br />
** ...<br />
<br />
=== April 29, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] <strike>Abstractions</strike> subworkflows status ([https://github.com/VisTrails/VisTrails/labels/subworkflows tickets])<br />
* [RR] matplotlib compatibility (2.1 & 2.2), [https://github.com/VisTrails/VisTrails/issues/1067 #1067]<br />
** RR to try and fix ALPS matplotlib modules<br />
<br />
=== April 22, 2015 ===<br />
'''Updates'''<br />
* 2.2.0 has been released!<br />
* Windows issue (via email from Ryan)<br />
** issue with manifest file (may be a new file in VTK6?)<br />
** Tommy has regenerated new Windows builds<br />
* Binaries, pypi, and conda released<br />
* [RR] Export as script<br />
* Python sources using VTK need to switch to SetInputData (users should be aware of this)<br />
<br />
=== April 15, 2015 ===<br />
'''Updates'''<br />
* Ready for 2.2.0 (apart from binary/deps issues)<br />
** Missing some libs (scikit-learn, tej, tdparser, SQLAlchemy+connectors<br />
** Windows: runvistrails.py is no longer used, so the PATH is wrong<br />
** Windows: pip is broken, but it probably wouldn't work anyway because of permission issues (disable this?)<br />
* Queries, upgrades and getPipeline() usage ([https://github.com/VisTrails/VisTrails/issues/1054 #1054])<br />
** Getting a pipeline with getPipeline() is not safe: it might return an invalid pipeline<br />
** This is used in many places throughout the code, like queries<br />
** Upgrading would require going through the controller, but that creates new actions<br />
* [TC] Avoid copying a module's output if it's used as input by exactly one downstream module ([https://github.com/VisTrails/VisTrails/issues/1060 #1060]) (useful for big numpy arrays you can update in-place)<br />
<br />
=== April 8, 2015 ===<br />
'''Updates'''<br />
* [DK] Merged RR's changes for output modules (1012 and 1013)<br />
** RR will merge remaining changes, then create v2.2 branch!<br />
<br />
'''Items to Discuss'''<br />
* BNL need numpy array to VTK image<br />
** Looks like VTK has some helpers for this<br />
** We will help if issues arise<br />
** Will contribute back to VisTrails package<br />
* Upgrade issue: [https://github.com/VisTrails/VisTrails/issues/1017 #1017]<br />
** Automatic upgrades should happen between versions of provided upgrades<br />
** Our existing upgrades work around this so it doesn't need to be 2.2.0<br />
* Corner-case VTK modules<br />
** No longer need VTKCell input port, so don't interfere with registry and API anymore<br />
** Still work weirdly, people probably shouldn't use them<br />
** But we have lots of clunky modules since we wrap the whole of VTK; some people might rely on this and know how to use them, let's keep them anyway<br />
** Ready for 2.2.0<br />
<br />
=== April 1, 2015 ===<br />
'''Updates'''<br />
* [RR] UV-CDAT: bugfixing for 2.2, long-term plans: implement scripting import/export in VisTrails, port to UV-CDAT<br />
* Possibly, try to move to regular VisTrails to use new features<br />
<br />
'''Items to Discuss'''<br />
* 2.2 release: nothing much is pending anymore, release next week?<br />
** [TE] vtkExporter classes [https://github.com/VisTrails/VisTrails/issues/1032 #1032]<br />
** [RR] Hiding upgrades in version tree might make it (or might be 2.2.1 so we can test it out) [https://github.com/VisTrails/VisTrails/issues/949 #949]<br />
** Output modules changes to go in<br />
* add note to documentation about order of parameters in VTK<br />
* add issue about exporter upgrades if not already there<br />
<br />
=== March 25, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* 2.2 release<br />
** Checklist on Github: https://github.com/VisTrails/VisTrails/wiki/2.2-Release-Checklist<br />
** Review output modules<br />
** RR has a couple more issues to fix<br />
** Ready to go -- sign app package for OSX? [https://github.com/VisTrails/VisTrails/issues/984 #984] We need access to the Apple Membership team<br />
<br />
=== March 18, 2015 ===<br />
'''Updates'''<br />
* [TE] VTK6 works<br />
<br />
'''Items to Discuss'''<br />
* [RR] Release v2.1.5 with backported tabledata?<br />
** MTA example needs updated tabledata (for JoinTables)<br />
** Google Maps package still not available<br />
* [RR] Work torward v2.2.0?<br />
** changes:<br />
*** new persistence<br />
*** API changes<br />
*** output module changes (upgrades?), maintain cells but try to upgrade<br />
*** not wrapping stuff<br />
*** VTK6? yes<br />
*** JobSubmission stuff?<br />
*** relabeling for upgrades [https://github.com/VisTrails/VisTrails/issues/949 #949]<br />
** makes sense, needs the tree view code to be updated, check selection<br />
** See [https://github.com/VisTrails/VisTrails/wiki/2.2-Release-Checklist 2.2 checklist]<br />
* Discussion of [https://github.com/VisTrails/VisTrails/issues/1016 #1016]<br />
** plumbing between outputs and output modes, how to define a mode that works for many outputs without writing for each output?<br />
<br />
=== March 11, 2015 ===<br />
'''Updates'''<br />
* [RR] persistent_archive done; merge? ([https://github.com/VisTrails/VisTrails/pull/755 #755])<br />
** note about the focus events for widgets<br />
** TE be aware of file_archive for future binaries that include persistent_archive<br />
* [TE] [https://github.com/VisTrails/VisTrails/pull/998 New VTK package] finished<br />
<br />
'''Items to Discuss'''<br />
* [RR] Unmark UV-CDAT/VisTrails as a fork of VisTrails/VisTrails? (see last week; decision needed)<br />
** RR email to JF about this<br />
** yes; email sent to Github<br />
* [RR] Switching order of output ports ([https://github.com/VisTrails/VisTrails/pull/1006 #1006])<br />
** added port specs are sorted at a separate spot (Module.*_port_specs properties) than those in the registry (which are sorted in the registry), but those two lists are just combined without respect to sort keys<br />
** need to determine whether the two lists should be merged or remain distinct<br />
** should make sure that order of input ports and output ports makes parallel connections for things with same order<br />
** DK suggests breaking backward compatibility here: workflows still run, can fix easily if a problem in an existing package.<br />
* [RR] Question about output modes ([https://github.com/VisTrails/VisTrails/pull/1007 #1007]), how to integrate in API ([https://github.com/VisTrails/VisTrails/pull/24 #24])<br />
** Should ImageFileMode be removed? ("image" is not a mode, "file" is)<br />
** ImageOutput missing?<br />
** Feel free to change how formats works<br />
* [TE] Test suite segfaults on Fedora 17 virtualbox. Install newer version? (Support ended in 2013)<br />
<br />
=== March 4, 2015 ===<br />
'''Updates'''<br />
* [TE] VTK wrapper<br />
** Works on VTK 5.10<br />
** Still need to test VTK 6<br />
** New general wrappers for python functions and classes <br />
<br />
'''Items to Discuss'''<br />
* [RR] Unmark UV-CDAT/VisTrails as a fork of VisTrails/VisTrails?<br />
** When [http://i.imgur.com/UTY2qfg.png filing a pull request] from UV-CDAT/VisTrails, VisTrails/VisTrails is selected by default<br />
** People keep forgetting to change the default ([https://github.com/VisTrails/VisTrails/pull/956 #956] [https://github.com/VisTrails/VisTrails/pull/968 #968] [https://github.com/VisTrails/VisTrails/pull/999 #999] [https://github.com/VisTrails/VisTrails/pull/1000 #1000] [https://github.com/VisTrails/VisTrails/pull/1003 #1003] [https://github.com/VisTrails/VisTrails/pull/1004 #1004] [https://github.com/VisTrails/VisTrails/pull/1005 #1005])<br />
** Only way to change that is to not have it marked as a fork of VisTrails/VisTrails<br />
** Github staff [https://gist.github.com/remram44/b364c24ef1e462ce6e2f can make that change for us]; should we do it?<br />
** Juliana: comments on this? visibility vs. convenience/annoyance for developers<br />
* [RR] What about [https://github.com/VisTrails/VisTrails/issues/991 #991]?<br />
<br />
=== February 25, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [TE] vtkviewcell for infovis support, can we unify with VTKCell?<br />
** need to test this<br />
* [TE] vtk wrapping<br />
** Mostly finished<br />
** VTK 5.10 produce incorrect results with old wrapping<br />
*** Old wrapper is based mostly on VTK 4<br />
*** Most vtk_examples affected<br />
*** [http://www.vtk.org/Wiki/VTK/VTK_6_Migration/Replacement_of_SetInput]<br />
*** should be able to upgrade from SetInput to SetInputData (need to drop GetOutput and replace with self ports)<br />
*** can we change vtkInstance to just return self and not wrap things<br />
** terminator example not working under 5.8?<br />
* How does VTK wrapping fit into general wrapping framework?<br />
* [RR] new persistence package<br />
<br />
=== February 18, 2015 ===<br />
'''Updates'''<br />
* [RR] New VisTrails API and IPython integration ([https://github.com/VisTrails/VisTrails/pull/24 #24])<br />
<br />
'''Items to Discuss'''<br />
* [TE] VTK wrapping<br />
** [https://gist.github.com/rexissimus/16b74693545b5e7c1035 Benchmarking vtk package]<br />
*** Old: 24.7 seconds <br />
*** New: 10.5 seconds (Except first time that adds 8 sec)<br />
**** The parsing that calls is_abstract (that tries to instanciate all vtk classes) is now only run the first time.<br />
**** get_items_from_sigstring takes 2 seconds, maybe we can use a lookup dict for already computed sigstrings?<br />
** Now using a general python function wrapper<br />
*** VTK classes are wrapped into python function that does not depend on vistrails<br />
*** VTK functions can be executed without vistrails<br />
*** The spec maps functions into vistrails modules, but can also describe wrapping <br />
*** A general python function wrapper that supports <br />
**** kwarg inputs<br />
**** single, list, dict outputs<br />
**** callback for progress reporting<br />
**** temporary file generator for using FilePool<br />
**** optional output generation<br />
*** Creating specs:<br />
***** Create spec by hand<br />
***** Auto-create spec outline (TODO) and manually finish it<br />
***** Dynamically create spec (VTK)<br />
***** Implement documentation wrappers (Can use scikitlearn wrapper to wrap numpydoc) (TODO)<br />
***** Classes as bad functions needs to be wrapped in new functions before they are wrapped. This is different for each package.<br />
****** Classes is hard: Like VTK, and matplotlib. Scikit-learn does still not wrap classes<br />
***** Spec diffing and patching could be done using code from matplotlib.<br />
** Still needs upgrades from old VTK package<br />
*** Is it possible to dynamically wrap functions, e.g, you see a SetFunc and just remove the 'Set' prefix. Or do you need to create a complete mapping?<br />
<br />
=== February 11, 2015 ===<br />
'''Updates'''<br />
* Update from Friday's meeting<br />
** discussed VisTrails internals<br />
** discussed wrapping<br />
*** xml discussion, hard to modify because tied to db code<br />
*** TE has made it possible to add the schema-defined attributes to the intermediate representation<br />
*** higher-level operations on the port specs<br />
** make sure the simple case works<br />
*** [JF] take a simple package with documentation and figure out what the base case for wrapping is<br />
<br />
'''Items to Discuss'''<br />
* [TE] VTK wrapping<br />
** Dynamic loading works<br />
** Reading XML is fast enough, but [https://gist.github.com/rexissimus/8272d3c776ccfa115d82 unserializing data is slow]<br />
** Working on patterns for patching<br />
** Matplotlib has many advanced patterns like argument ordering, nested arguments, alternateSpecs, output types.<br />
*** Having all this in a general wrapper might confuse users?<br />
** [RR] Delay module (except for identifiers) until you need it---e.g. don't deal with port specs, etc. until necessary<br />
* Scripting Support [https://github.com/VisTrails/VisTrails/pull/950 #950]<br />
** [RR] Issue with getting code from modules<br />
** Design a simple solution<br />
** [JF] Couldn't you use modules as black boxes without conversion, just to call into modules/subworkflows easily from e.g. IPython?<br />
*** [RR] This is a job for the API, and a very separate use case. See [https://github.com/VisTrails/VisTrails/pull/24 #24]<br />
<br />
=== February 4, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* Wrapping<br />
** Format to use? Currently XML (like current matplotlib)<br />
*** JSON and YAML have simple "to python dictionary" methods<br />
*** But don't stream<br />
*** YAML a lot easier for humans<br />
** [DK] vtk-new-package also changes parameter names, creates enumerations<br />
*** intermediate schema needs to be extensible<br />
*** packages will want to store there specific infos for compute() method generation<br />
*** also might have specs-altering info, like matplotlib's alternateSpec<br />
** representation to code , registry already has schema for some aspects<br />
** [RR] We might want to see if Module subclasses can be created lazily<br />
*** no need to create all the classes just to register them in the registry and never actually use most of them<br />
*** future effort<br />
* [RR] Where should VisTrails packages live?<br />
** tej installs as 'vistrailspkg.tej', TE installed it as 'userpackages.tej'<br />
** Currently, standard packages are 'vistrails.packages.', user packages are 'userpackages.' and packages loaded through pkg_resources might be anything<br />
** [RR] Use 'vistrailspkg.' everywhere?<br />
** Long-term effort to simplify package distribution/installation (and have VisTrails get them automatically?)<br />
<br />
=== January 28, 2015 ===<br />
'''Updates'''<br />
* T. Caswell to come visit on Fri 6 to discuss wrapping work<br />
<br />
'''Items to Discuss'''<br />
* [TE] New VTK wrapping<br />
** Current code by DK seems a good deal faster<br />
** Generates XML that can be patched/tweaked, generates Python code from it, VisTrails only loads generated Python code<br />
** RR would rather have VisTrails load intermediate representation (= XML) directly, wants to make sure this is not slower<br />
** The goal is to turn the intermediate step into something generic that would be used for every wrapped package (vtk, numpy, matplotlib, sklearn, java) instead of each having its own<br />
** TC has its own code at [https://github.com/Nikea/VTTools/blob/master/vttools/wrap_lib.py github:VTTools] which parses numpy docstrings and generates modules, doesn't yet handle classes or persist anything<br />
* Web crawler<br />
** Right now, TE starts jobs for "start crawler", "stop crawler", "install classifier"<br />
** RR would rather have the crawling be a job as far as VisTrails and tej are concerned<br />
** The whole thing would be one pipeline: load examples, train classifier, start crawler [check for job, kill previous one, upload model, start processes], get snapshot, visualize<br />
*** Need some support in tej and job submission system: long-running jobs, stop a job (wait for it to finish?), restart a job even though results are cached<br />
<br />
=== January 21, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* [RR] Unified wrapping method discussion [https://github.com/VisTrails/VisTrails/issues/991 #991]<br />
** TE to work on reusable method with intermediate representation, starting with VTK<br />
* [RR] Examples for scikit-learn: JF has an old example using Weka with parameter exploration (not currently in source tree)<br />
** AM's examples are enough<br />
* [AM] scikit-learn package is done, merge it in? [https://github.com/VisTrails/VisTrails/pull/955#issuecomment-70286788 #955]<br />
** RR will merge<br />
* [RR] What should copyright headers say? [https://github.com/VisTrails/VisTrails/pull/994#issuecomment-70173935 #994]<br />
** Let's keep everything in there: Utah/Poly/NYU<br />
<br />
=== January 14, 2015 ===<br />
'''Updates'''<br />
* [TE] Working on classifier<br />
* [RR] Scripting integration, work in progress<br />
<br />
'''Items to Discuss'''<br />
* [RR] Unified wrapping method discussion ([https://github.com/VisTrails/VisTrails/issues/991 #991])<br />
** Let's talk next week, [AM] and [DK] are not here<br />
<br />
=== January 7, 2015 ===<br />
'''Updates'''<br />
<br />
'''Items to Discuss'''<br />
* make sure that we address critical issues, questions, and pending review branches in a timely manner<br />
* scripting support<br />
** [RR] no issues if we want to just keep annotations in the generated code to allow the link back to a workflow<br />
** [RR] can translate from workflow to script, working on script to workflow<br />
** will work for parameter value changes, structural changes require changes to the annotations<br />
** need to publish best practices here<br />
** would be cool to do looping in scripts (easier interface than with workflows)<br />
* notebook support (convert form notebook to workflow)<br />
** RR will sync with FC on this<br />
* Issue with console in built-from-scratch<br />
** [TC] iPython rearranged some of the completion stuff in 2.2 and 2.3<br />
** binary has old version of iPython -> 1.0.0, should we update?<br />
* [TC] automated wrapping of numpy and scipy<br />
** discovered a bunch of malformed documentation in numpy and scipy<br />
** has github repo for vistrails tools<br />
** example modules wrap a bunch of R stuff (not baked in, just how things are)<br />
** will be pushing wrapping logic up<br />
** port names forbidden (window and domain)<br />
** have an import hook to get from yaml directly to VisTrails Modules<br />
** should work for any python modules with well-formed numpy docstrings.<br />
* [Action] should make it clear in documentation that Constant now means serializable not that the value doesn't change (e.g. List)<br />
* [TC] might be interesting to try to build components of matplotlib and accumulate in figure (long-term project, but thinking about how this might work)<br />
* [TE] build and build scripts<br />
** completely automatic, buildbot<br />
** need to set the build machines for the environment we want for the binary<br />
** would virtualenv work here?<br />
** [TC] anaconda can pin versions, potential path to test different configurations<br />
** Q: upload nightly binary builds? A: makes sense, make sure they are well-labeled<br />
* sourceforge stats: e.g. http://sourceforge.net/projects/vistrails/files/vistrails/nightly/vistrails-src-nightly.tar.gz/stats/timeline?dates=2014-01-01+to+2015-01-07<br />
* package issues (see Remi's message)<br />
* [TE] Scope of tej<br />
** Support single ssh commands?<br />
** Queue can be used as a remote machine (crawler is using queue.call*)<br />
* SourceForge stats: http://sourceforge.net/projects/vistrails/files/vistrails/nightly/vistrails-src-nightly.tar.gz/stats/timeline?dates=2014-01-01+to+2015-01-07<br />
<br />
== Older meetings ==<br />
* [[Development/2014]]<br />
* [[Development/2013]]<br />
* [[Development/2012]]<br />
* [[Development/2011]]<br />
* [[Development/2010]]</div>Remi