VistrailsWiki - User contributions [en]

https://www.vistrails.org//api.php?action=feedcontributions&feedformat=atom&user=Mates VistrailsWiki - User contributions [en] 2026-05-05T07:34:07Z User contributions MediaWiki 1.36.2 https://www.vistrails.org//index.php?title=Development&diff=3757 Development 2010-12-14T20:08:56Z

<p>Mates: /* Dec 14, 2010 */</p> <hr /> <div>== Roadmap ==<br /> <br /> <br /> == Weekly Meetings ==<br /> <br /> === Dec 14, 2010 ===<br /> <br /> * [ES] Unit tests<br /> * Data publishing with CrowdLabs <br /> * [DR] Subworkflow upgrades: update--is it a wrap?<br /> * [TE] [[Data/provenance browser]]: update<br /> * PythonSource editor<br /> * Lack of global variables and bookmarks<br /> * [PM] Merging master and crowdlabs git branches. How should we merge the crowdlabs branch into the master branch?<br /> <br /> === Dec 7, 2010 ===<br /> <br /> * [DR] Subworkflow upgrades<br /> ** Made sure that we don't rewrite actions but create new actions that show where subworkflows are upgraded<br /> ** Document what is implemented and how it works (at high level), also what isn't yet implemented<br /> <br /> * [TE] Errors and web services <br /> ** allow user to decide to only display errors to messages window instead of also showing popup windows<br /> ** Allow add/delete of web services from GUI without going to preferences<br /> ** Qt dependencies for error messages gone<br /> <br /> * [PM] Some specific questions<br /> ** Question: Should v1_0_1 MySql schemas be upgraded to v1_0_2?<br /> ** db_connection.begin() being removed, stale results<br /> ** memory size is too big for the field size<br /> ** [DK] check on schema changes, see what is important for changes<br /> <br /> * Question: do we display 'warnings' or only errors?<br /> ** Need to write ModuleErrors (maybe make ModuleWarnings) to the messages (debug) as well as adding as tool tips<br /> <br /> * Some suggestions/requests by Matthias:<br /> ** Support the grouping of vistrails and files into projects<br /> *** Annotation here?<br /> ** Ability to set global parameters/bookmarks=workflows (e.g., a filename or path)<br /> ** Archive results -- create a big tar/zip file with all vt and results<br /> ** Ability to add a vt: if a node is deleted, all data associated with that should also be deleted<br /> ** Create a test suite: run a workflow, save and archive results (user could specify which results should be saved)<br /> <br /> * [[Data/provenance browser]]<br /> ** We need a better mechanism that allows users to search/query for workflows and provenance, for example search by project, parameter names and values<br /> ** One way to view this is to allow the creation of mappings between the vistrails/projects into different structures---like "smart folders", e.g., /projectN/vt1/wf3/exec23/O1....<br /> ** Support editing and synchronization in both ways (if edit smart folder, should update vt)<br /> ** Should be able to script the use of the browser, e.g., to test a new version of a workflow against a saved test suite<br /> ** Scientists like a directory structure and can go browse there like they browse files with finder/explorer<br /> ** Keep mappings from our datatypes to file structures<br /> ** Need an exportable archive (to send with a paper, allow browsing outside of VisTrails)<br /> <br /> * On merging master and crowdlabs git branches<br /> ** some updates on the crowdlabs branch need to be merged/patched into the master branch<br /> ** how should this be done?<br /> <br /> * PythonSource editor<br /> ** modal or multiple window editors?<br /> <br /> * Lack of global variables and bookmarks<br /> ** Problem is that aliases don't change workflow specification<br /> ** Need to store these values in the execution log<br /> ** Keep alias dictionary for everything<br /> ** [PM] Not a need for global variables above the single workflow level<br /> ** This already exists at the conceptual level---one module with connections to other modules, this just is cluttered<br /> <br /> * Unit tests<br /> ** [ES]: I ran the test suite on current master branch and 207 tests are executed of which 11 fail. Some of the tests are failing because console_mode_test package can be imported anymore (this is a package created specific to run console_mode tests).<br /> ** Most of the other problems are issues with<br /> <br /> InvalidPipeline: Pipeline cannot be instantiated:<br /> Missing version 0 of package <package><br /> <br /> * Data publishing with CrowdLabs<br /> ** Using git to manage data between local and server data stores might be too messy.<br /> *** Does one create a new git repository for each vistrail on crowdlabs?<br /> *** The local persistent git repo holds all persistent data used on that machine. How does one push/publish just one data product using git?<br /> *** Can anybody think of a plausible git workflow to do this?<br /> ** Perhaps we should just look into adding directory support for RepoSync module along with a better data management UI.<br /> ** Are there OS independent file transfer protocols better than HTTP? Something like rsync might work nicely, but windows support is limited.<br /> <br /> === Nov 30, 2010 ===<br /> <br /> * Subworkflow Upgrades (Daniel)<br /> ** Should be completed finished tonight <br /> ** Local subworkflow upgrades should be working now<br /> ** Need to test these after the checkin<br /> ** Using a dummy controller<br /> <br /> * Assistant for control package (Daniel)<br /> ** Look at examples<br /> ** Try to make an assistant to modify workflow using currently selected modules as looping group<br /> <br /> * [[Improve error handling/reporting]]<br /> ** Tons of message boxes pop up when a workflow has a bunch of errors, UI response is bad after this...<br /> ** Update existing error messages<br /> ** Show Details rendering on Mac<br /> ** What to do when multiple errors occur in a single event?<br /> ** New dialog that shows "next message" but allows user to dismiss messages or view all in the messages window<br /> ** Color coding for messages: use gui/theme.py to configure: maybe try black for warning, gray for log level messages<br /> <br /> * Data and workflow browser (Nivan will give a status update)<br /> <br /> * Saving Explorations (Feature Request from ALPS)<br /> ** We already save parameter explorations with the workflow in the version tree. As is, we save only the last exploration that was done for a given workflow. If you ship the vistrail, the saved parameter explorations will be there. Is this sufficient?<br /> ** [MT] Yes, that takes us a long way. Could one add a button in the explorations to save a certain set of explorations as a new version, like you do with the camera angle in the spreadsheet? That way one could store more than one set of exploration parameters without forcing a new version to be created with every change?<br /> ** Eventually we should provide a selector in parameter exploration<br /> ** Fastest to just allow a null action new version (a bit clunky) but wouldn't require too many changes<br /> <br /> * Clean up/prune persistent directory<br /> ** What are the concrete use cases for this?<br /> ** [MT] Here are two concrete use cases that appeared over the last week:<br /> *** a project is done, the important results copied to an archive and I want to get rid of all its intermediate files, etc., to save space - but not those files associated with other projects. Or, similarly, I'm done testing some workflows, want to remove all (partially buggy or irrelevant) files, and then start with production runs.<br /> *** there was a problem in a workflow(e.g. a bug in HTTPFile module with binary files or a bug in some user code). I want to force a certain persisted directory to be "forgotten", so that I can repair the wrong files. Bumping the version number of the ALPS package would be one way, but that is overkill since it would immediately invalidate all persisted files. Removing the whole persistence directory is just as radical.<br /> ** just testing simple tutorials with small files leaves me with 2 GB of persistent data after 10 days, in 142 persistent directories or files. I need some way to clean this up that is less radical then removing all persistent directories and files. This could be done, by e.g. removing all those which do not occur in tagged or leaf workflows in a certain set of vistrails. Or marking those which occur in certain vistrails with certain labels, and then being able to prune all those with or without some label/tag. Another option might be having the ability to choose from various repositories.<br /> ** [DK] follow up on this<br /> <br /> * Improvements to VisTrails server communication with clients to better handle errors/crashes (Phillip)<br /> ** work in progress<br /> <br /> * Data Publishing with crowdLabs. (Phillip) <br /> ** Do we want data publishing capabilities in VisTrails and crowdLabs? '''YES!!!'''<br /> ** If so...<br /> *** Currently limited data provenance for data. HTTPFile module lacks meta data, RepoSync module has poor usability.<br /> *** Persistent Data works for experimental data. Can we use it for publishing data?<br /> **** Publishing data or storing data on the server and pushing back and forth?<br /> **** What is the current scheme for identifying data on the server?<br /> **** Do we need to do better than the ids, versions, hashes that the peristence package provides?<br /> *** Do we want to utilize data publishing software like [http://thedata.org/home The Dataverse Network] or maybe [http://www.fedora-commons.org/ Fedora Commons]?<br /> ** [ES] Feature request: Can RepoSync support a directory of files?<br /> *** [PM] It can't, and I'm not sure how add it. I also don't really like the current RepoSync interface. It's not obvious how your data is being handled, hence me bring up data publishing, so hopefully we can come up with something more usable/robust.<br /> ** persistence used more for exploratory<br /> ** want to focus on more archive<br /> ** move from exploratory data to published<br /> ** Dataverse: essentially a site to curate data, archive data, host their own site, each university has their own server, linked together<br /> ** persistent store integration: maybe use git to do this, can add to local repositories to move to git<br /> ** use some of the dataverse: standard hash schemes, etc.<br /> ** privacy: can users push workflows without exposing data<br /> ** permissions for who can check things out: download permissions, upload permissions<br /> ** look into setting up persistent store on <br /> <br /> * DisplayWall (Wendel)<br /> ** new version of vistrails receives rotate/scale messages much more slowly<br /> ** using the vtk version that is already there<br /> ** comparing VisTrails 1.2 with nightly src<br /> ** test when the messages are received on the clients<br /> ** [ES] Changed something in the VTKCell, using the Qt interactor from vtk now, just uncomment this (vtkcell.py file)<br /> ** some commands are missed; check why this is happening<br /> ** preferences switch? 2x2 not 2x3<br /> ** check what modifications exist in the client code that's not exactly 1.2?<br /> ** new version of vtk package changed some things<br /> ** update vtk upgrade: resetcamera and addrgbpoint => _# versions<br /> ** Demos on displaywall: Wendel has to keep both VisTrails running and stop work when visitors stop by unannounced<br /> <br /> * Relative paths in workflows, how to deal with them? (Emanuele)<br /> ** [ES] Updated all examples to use HTTPFile for these<br /> ** some of these had to change completely (color widget), kept tree the same format<br /> <br /> === Nov 16, 2010 ===<br /> <br /> * Data and workflow browser (Nivan)<br /> ** Check into git branch (Dave)<br /> ** Add data support to the vistrails, workflows, and execs already exposed in the browser<br /> ** Add thumbnails to the browser<br /> ** Bittorrent support?<br /> <br /> * Upgrade issues -- a wrap? (Daniel)<br /> ** Fixing some issues with namespaces<br /> ** Upgrades for subworkflows that are included with packages, need to save these upgrades with the vt file as a "local abstraction" to ensure provenance. May be able to tie these back to the package versions once the package is upgraded.<br /> <br /> * Test Suite<br /> ** Matthias has his own test suite now<br /> ** We should make sure that our tests are up-to-date and run them before releases<br /> <br /> * Web services on users' guide (Tommy)<br /> ** Users' guide needs only a few changes. One of the changes should be a note making it explicit that we are using a new library and the old one was deprecated.<br /> ** Do we want to update the 39-page guide as well? For now, just update the users' guide. Don't distribute this guide. Just include it in a zip file that includes the old version.<br /> <br /> * [[Improve error handling/reporting]] (Tommy)<br /> ** print statements, has a new window<br /> ** add a menu item to view message window and also keep all messages from the session<br /> ** look into writing QMessageBox errors and warning to message window<br /> ** write all messages to a single window, write critical messages in red?<br /> ** have a utility method that encapsulates the gui error display and the core.debug display (if not running gui, only write to core.debug)<br /> ** don't display stack trace in a qmessagebox? -- try to emulate the invalid pipeline with "Show Details" button.<br /> ** make it easier for users to report errors by having a "copy" or "send to developers" button to report the details of an error.<br /> <br /> * DisplayWall (Wendel)<br /> ** What examples should be in the SVN? If we decide to keep some examples in the svn, some of them use data that should be along.<br /> *** Indicate progress of HTTPFile<br /> *** Had to upgrade examples to work with new examples of VisTrails<br /> *** VisTrails can send progress information with another thread like crowdlabs can do<br /> *** Each machine has a .vistrails and needs to download own version of data, make a single .vistrails for all machines?<br /> *** All slave machines can use the server as a cache?<br /> ** DisplayWall Client on the Apple Store? I dont like the idea of sharing the client source code. And putting the app without login/password will allow anyone to access our displaywall.<br /> *** Add a password for this<br /> *** Do user management?<br /> <br /> * VTK package:<br /> ** Have two ResetCamera modules now<br /> ** Need to write upgrade code<br /> ** Only with vtk 5.7 right now<br /> ** Encode library version in the package version?<br /> ** For reproducibility, we probably want to stick to specific versions<br /> ** We still need some ability to run new versions of vtk for testing, new features, etc., too...<br /> <br /> * Relative paths in workflows, how to deal with them? (Emanuele)<br /> ** Should we use dataDirectory?<br /> ** Change examples to not use hard-coded files?<br /> ** Problem with workflows, not data files<br /> ** On Windows, use ../examples<br /> ** On Mac, use ../../../examples<br /> ** dataDirectory not used, I think<br /> ** create "$VT_EXAMPLES"?<br /> ** problem is using new versions of VisTrails, data may change so referencing old location will fail<br /> ** don't allow users to change this variable<br /> ** convert to use HTTPFile<br /> <br /> * Data files and the GIT Repository (Emanuele)<br /> ** make a separate repository or use the submodule git functionality<br /> ** go with submodule<br /> <br /> * Next release: What still needs to be done?<br /> ** high priority from ALPS list<br /> ** Dan's fixes to the upgrade workflow<br /> ** Date for release? Official release in beginning of December for ALPS<br /> ** All developers should run test suite on push<br /> ** Could make this somewhat automatic, run with nightly release script and email -dev list<br /> ** Could have different classes of tests, make sure that critical ones pass<br /> ** Tests have a bunch of errors currently<br /> <br /> * Saving explorations:<br /> ** Keeps track of the latest exploration for each version<br /> ** Check with Matthias what the requirements for this are<br /> ** Have way to send just specification to a server (run this workflow with this parameter exploration)?<br /> ** Can we have an xml spec that specifies the parameter exploration (export parameter exploration)<br /> ** Currently persisting via xml and then converting to string so we already have an xml serialization<br /> ** Dan wrote API that takes xml string and populates the param_exp gui with the appropriate values<br /> ** Need to back the GUI state with core state to enable short-circuiting the gui<br /> <br /> <br /> * Box input ports should be colored (e.g. black) if they are already set by an internal method. Another color (e.g. red) could be used to indicate mandatory input ports.<br /> ** Currently, non-optional ports are square, shown. Optionals are circles and must be enabled<br /> ** Add a third category for mandatory, problem is backward compat.<br /> ** [ES] Colors are already overloaded, not for ports, use shading instead?<br /> <br /> === Nov 9, 2010 ===<br /> <br /> * FAQ<br /> ** As we reply to users queries, let's add the question and answer to the FAQ!<br /> ** Just a reminder<br /> <br /> * Caching of File module<br /> ** There was a message from a user who got confused because this module is cached by default (and silently). And in his application, since the file actually changed in between runs, he did not see the 'expected' result.<br /> ** Should we keep a hash for files and check whether they changed?<br /> ** Should we no cache File by default?<br /> ** [DK] File is supposed to have a special signature computation that detects changes in the file contents so this may be a bug instead. It does work, it seems to be an issue with the create_file<br /> ** [ES] In this particular case, the file was empty. He deleted the file outside VisTrails and executed it again. As he set create_file to True, he was expecting the file to be created again.<br /> ** [DK] Add random salt to hash signature when the file doesn't exist and create_file is True<br /> ** [JF: DONE] Juliana will edit the FAQ to note that the signatures are dependent on modification times<br /> <br /> * Claurissa will demo different ways to visualize the version tree<br /> ** weighting on various criteria for displaying nodes (importance from session, user, tags, etc.)<br /> ** other modes for viewing the versions (lists or timelines)<br /> ** often see only linear trees, teaching or thinking other than tree?<br /> <br /> <br /> <br /> * Upgrade issues (Dan)<br /> ** Vanilla upgrades should be in today's commit<br /> ** Added code so when doing upgrade to check whether the latest code is upgraded<br /> ** Two types of upgrades: (1) User changes the subworkflow, (2) Modules are out-of-date and need to be upgraded.<br /> ** Add higher-level features like "Upgrade All" for (1), "Import Changes to My Subworkflows", etc.<br /> ** Want a 1-1 mapping between unique subworkflow id and a subworkflow file (even if we copy the same subworkflow to different vistrails)<br /> <br /> * Offscreen bug (update from Huy)<br /> ** Mac issue when switching to Cocoa<br /> ** Huy added a QWidget for each of the render windows that is created, so should be fixed.<br /> <br /> * Gesture support<br /> ** Someone on Windows check the pinch gesture support; it doesn't work for Matthias under Parallels<br /> ** VTKCell is working now<br /> ** Cell dependent, panning now maps to middle button, pinch on the base widget<br /> ** What are Matthias's requirements here? Does touch gestures in the cells matter or only in the vt/wf views<br /> <br /> * Thumbnails: [JF: this needs to go to trac, for 2.0?] [ES: 2 tickets were created for 2.0]<br /> ** Resolution? -- [ES] problem is the size of the vistrail, but this can be configured<br /> ** Can we export to PDF instead of PNG? [ES] cannot display the thumbnail as a PDF in Qt?<br /> ** Can we have the option to save higher-res or PDF versions of the thumbnails<br /> ** Can we have preferences that allow users to save a higher-res or PDF version, maybe a checkbox<br /> ** Maybe associate a high-res version with a spreadsheet cell so that we can save a high-res version on demand<br /> ** high-res version associated with a version but allow user to initiate action from spreadsheet<br /> ** can we have thumbnails dependent on camera position [ES] doesn't like, user won't see exactly that image upon re-executing<br /> ** compact vistrail option by removing thumbnails<br /> <br /> * Tommy has rewritten the Web services module as well as updated all of the examples<br /> ** Server for one of the examples seems to be down<br /> ** Need to update the manual<br /> <br /> * [[Improve error handling/reporting]] (Tommy)<br /> ** Maybe display the intialization messages with the splash screen?<br /> ** Debugging levels have no way to access debug level<br /> ** Tommy takes first cut at trying to determine how to map print statements, notes those he is unsure of<br /> ** Developer guidelines for debug usage, use them instead of print statements in the futur<br /> <br /> * Merge functionality (Tommy) [JF: Cool!]<br /> ** Can now merge an existing vistrail into your own vistrail using menu item<br /> <br /> * Improvements to VisTrails server communication with clients to better handle errors/crashes (Phillip)<br /> ** [PM] I haven't had a chance to implement anything yet.<br /> <br /> * Data Publishing with crowdLabs. (Phillip) <br /> ** Do we want data publishing capabilities in VisTrails and crowdLabs? '''YES!!!'''<br /> ** If so...<br /> *** Currently limited data provenance for data. HTTPFile module lacks meta data, RepoSync module has poor usability.<br /> *** Persistent Data works for experimental data. Can we use it for publishing data?<br /> **** Publishing data or storing data on the server and pushing back and forth?<br /> **** What is the current scheme for identifying data on the server?<br /> **** Do we need to do better than the ids, versions, hashes that the peristence package provides?<br /> *** Do we want to utilize data publishing software like [http://thedata.org/home The Dataverse Network] or maybe [http://www.fedora-commons.org/ Fedora Commons]?<br /> ** [ES] Feature request: Can RepoSync support a directory of files?<br /> *** [PM] It can't, and I'm not sure how add it. I also don't really like the current RepoSync interface. It's not obvious how your data is being handled, hence me bring up data publishing, so hopefully we can come up with something more usable/robust.<br /> <br /> === Nov 3, 2010 ===<br /> <br /> * Upgrading subworkflows (Dan)<br /> ** add updated versions to subworkflow vistrail and push to registry<br /> ** fixed bug with version/descriptor redirects<br /> ** key remaining task is to replace the box representing the subworkflow in the top-level workflow to reflect the change to the upgraded version of the subworkflow<br /> ** many corner cases, but we should get the vanilla upgrade (working) to git<br /> ** need to ensure that each subworkflow file has its own uuid; if a subworkflow exists in two different vistrails, it should have a different uuid (namespace).<br /> ** otherwise, we can get crosstalk where one vt can update the subworkflow from another vt (we can propgate changes to other vts via merges, but this should be a user choice...)<br /> ** enhancements:<br /> *** allow users to merge subworkflows from a file into their own subworkflow (to incorporate outside changes)<br /> *** latest version is the most recent non-upgraded version; if that version has an upgrade, use the upgrade<br /> ***'''Next step: work on usability for controlflow package; explore Dave's idea of an assistant'''<br /> <br /> * Offscreen bug<br /> ** Huy is looking into this<br /> <br /> * Crash due to issues in loaded user packages (Tommy has fixed this)<br /> <br /> * Web services (Tommy)<br /> ** Package is completed, and all but one of our examples are working<br /> ** '''Tommy will test if the Web services will work when there is a proxy'''<br /> <br /> * Improve error handling/reporting (Tommy)<br /> ** we should have a single point for all error messages to pass through that is linked to both core.debug and GUI elements that display error messages. <br /> ** the goal is to implement a function that will be the single point for error messages, and that work without emitting GUI signals in core. As Huy suggested, we should have a GUI-wrapper which will be a no-op when the GUI is not instantiated---this will get rid of some of the pyqt dependencies (at least for the error messages).<br /> <br /> * Improvements to VisTrails server communication with clients to better handle errors/crashes (Phillip)<br /> ** Currently the VisTrails servers consists of: a single-threaded instance with GUI; multiple GUI-less threads<br /> ** We will extend the server API to allow clients to check the server status. The server will provide a separate socket and the client will be able to 'ping' the server; if the server is working properly, it will respond; if it does not respond, the client will have a time out<br /> ** The API will also support the ability to kill both the GUI-less threads and the single-threaded GUI instance. We can then restart the server components using Emanuele's new script.<br /> ** We should also allow users to set a per-workflow time-out; if the workflow execution takes longer than the pre-defined threshold, VisTrails will abort the execution<br /> <br /> <br /> * Update on fixing VTK Package (Wendel)<br /> <br /> ** It seems that the ParaView package has the same issues.<br /> ** '''Wendel has already checked in the new wrapper into the trunk; Emanuele will test it'''<br /> <br /> * Synchronizing VisTrails and ALPS releases (Emanuele)<br /> ** VisTrails latex package was extended so that python is no longer required. If python is not present in the system, instead of issuing a request to execute the workflow and retrieve the image from the Web, a local, previously saved image will be used. It is now also possible to embed the images of workflows into the latex file. Documentation about these and other features are currently in the README file and examples (example.tex) provided with the latex package.<br /> ** Windows and Mac beta binaries already include ALPS<br /> ** Are we distributing ALPS as part of VisTrails then? This seems backwards. ALPS should VisTrails as part of their stuff, no?<br /> *** '''We will have separate binary distributions: one with and one without ALPS'''<br /> <br /> * Is a multi-touch interface to VisTrails possible now that it's supported by Qt? (Phillip)<br /> ** Yes, Qt 4.6+ supports multi-tocuh --- '''Huy will look into this; the goal is to make this work both for the workflow builder/tree view and the spreadsheet'''<br /> ** We still need to figure out what makes sense for multi-touch, however, do we have requirements from ETH-Zurich?<br /> *** We want to map the zoom, pan, click, etc to multi-touch, so that VisTrails can be intractable on a multi-touch screen without a mouse<br /> <br /> * What is the status of the Vismashup i{Phone|Pad|Touch} app? (Phillip)<br /> ** If you mean mac binary, there's an alpha version here: http://www.sci.utah.edu/~emanuele/files/vismashup <br /> ** Do you mean iphone app? [Phillip: yes]<br /> ** '''Wendel will look into this'''<br /> *** One of the issues is how to effectively handle images that are larger than the memory on the iPods and iPads---we need to process these on the server or use a model that allows the image to be manipulated on the client<br /> *** We also need to connect the app with the crowdlabs server, so that the app can get the list of mashups<br /> *** Phillip will investigate the feasibility of working directly with JavaScript and bypass flash<br /> <br /> <br /> <br /> === Oct 19, 2010 ===<br /> <br /> * Update on Trac and Roadmap (Emanuele and David)<br /> ** Trac is now linked to git repository<br /> ** Roadmap on Trac has been cleaned up so we can hide completed milestones<br /> ** Tickets are being reassigned/revisited. Many tickets (24) are not associated with a milestone yet https://vistrails.sci.utah.edu/report/3<br /> * Synchronizing VisTrails and ALPS releases<br /> * Web services support (Tommy)<br /> ** discuss the interaction between the Web Services package and upgrades<br /> * Subworkflows update (Daniel)<br /> * Testing of new module drawing (Erik)<br /> * Aliases<br /> ** Currently, an alias is stored on a parameter. We need aliases to be stored at a higher level so that changing an alias is not a change to a parameter. The one issue is that there is a link between parmaeters and aliases in that an alias can only exist for versions that have the specified parameter. We might also have two versions where the alias points to different parameters. We could just store aliases as a root-level workflow element so that the set of aliases is versioned corrected.<br /> ** In the current implementation, the aliases parsed from the parameters are stored in a dictionary in the workflow, so storing aliases as a root-level workflows element might be the way to go --[[User:Emanuele|Emanuele]] 19:50, 15 October 2010 (MDT)<br /> * Error logging<br /> ** Suggest that we have a single point for all error messages to pass through that is linked to both core.debug and GUI elements that display error messages. This should improve our error handling significantly<br /> * PyQt dependencies in core<br /> ** Can we get rid of PyQt dependencies in core (and db)? This would probably require moving the signals/slots that we currently use to a similar Python implementation which shouldn't be too difficult. This would also require the specification of configuration widgets not as classes but rather some text that can be used to import the GUI elements only when using the GUI code. However, what would happen with workflows that run in command-line mode but require some graphical output?<br /> * Ports<br /> ** One annoying part of the current VisTrails model is that users are not given any visual indication that a port value is already set by a function or vice versa by a connection. In addition, we have no way to specify or enforce cardinality on ports at design time. It seems like we could allow developers to specify when a port should not be connected to more than one value (via a function or a connection), and give some visual feedback when a port has already been specified in one manner.<br /> <br /> ==== Meeting Notes ====<br /> <br /> * Bug regarding userpackages at initial vistrails startup causing startup to fail.<br /> ** Still an issue, ticket still open as a 1.6 milestone<br /> * Copy-paste bug<br /> ** Shortcut not working on Mac. Focus makes shortcut try to copy pipeline.<br /> ** Current fix appends to clipboard as users type - this is not a good fix.<br /> * Synchronizing ALPS and VisTrails releases<br /> ** 1.6 scheduled for Dec. 1<br /> * Webservices<br /> ** New library being used, but it's very low level.<br /> ** Simple types may cause problems with some web-services using complex or XML-based types.<br /> ** Need to make sure that this library can handle at least MOST of the web-services out there.<br /> * Sub-workflows<br /> ** Daniel's changes seem functional and very slick.<br /> ** A little more work on it seems necessary.<br /> ** Need some easy GUI mechanism to delete a subworkflow.<br /> ** Is the exclamation point in the module draw the best way to handle this? Make a tool-tip to explain it.<br /> * VTK Wrapping<br /> ** Changes to VTK Python wrapping is causing headaches - particularly in backwards compatibility.<br /> ** Method typing has changed to expect things like "List" or "Vector" - makes things hard on the user when defining these inputs.<br /> ** Removing List and Vector types and replacing them with dynamically generated versions from Tuples.<br /> * Aliases - Agenda pushed until after Vis.<br /> * Error Logging - Unify some error handling to improve how exceptions are dealt with. More on this after Vis.<br /> * Refactoring out PyQt dependencies from Core/Db/etc - Need to move all the signals/slots into GUI elements. Questions come when a workflow uses GUI elements as inputs. - More after vis.<br /> * Ports - We need a way of showing how a port is specified - function vs. input port. Cardinality must be established in these cases. Ordering on multiple connections should be handled in some way - whether it's just a documentation issue or a more fundamental one.<br /> ** Should we have a tutorial mode that bring up tooltips when someone does something new as if it were a live tutorial.<br /> <br /> <br /> === Oct 12, 2010 ===<br /> <br /> * Trac and Roadmap (Emanuele): Emanuele suggested we should create a development roadmap and make it a permanent item on the agenda. The idea is to go through the Trac tickets and use them to build the roadmap that would be made ''public''.<br /> ** go through trac and sort by priority<br /> ** add other suggestions to roadmap as well<br /> ** admins can add milestones via the admin tab<br /> ** ''David will assign tasks on list; those assigned need to follow up and check and fix them''<br /> <br /> * Update on [[PythonSource error reporting and logging]] (Tommy)<br /> ** ''Need to make sure that all messages are printed through debug.X, so that the debug level (verboseness in preferences) is correctly used''<br /> ** Can cut/copy/paste now<br /> ** Can we prevent the delete from happening?<br /> ** Phillip noted the spyder project has an interactive python console: http://packages.python.org/spyder/<br /> ** Can press enter to get to the end<br /> ** Need to check if this works or if we need to change<br /> ** Now have ability to see stack trace from the GUI (triangle menu)<br /> ** ''Suggest using a dialog to display trace when this item is selected instead of printing to console''<br /> ** Also saving stack trace to the execution log<br /> ** Also printing debug information to vistrails log<br /> ** ''Add trac item to fix prints to use debug''<br /> ** ''Try to fix core.debug to have gui.debug to eliminate Qt dependency''<br /> <br /> * Update on subworkflow (Daniel)<br /> ** Notification when subworkflow is outdated triggered<br /> ** new_abstraction signal from registry in addition to new_module signal so that we can check if we need subworkflows to update<br /> ** How to display the possible upgrades (can be upgraded, can be upgraded but may break, etc.)<br /> <br /> * New utility added by Carlos (currently at scripts/module_appearance) to draw fringes visually, which generates output that can be pasted directly into the add_module call. <br /> ** ''Erik will test this on Mac and Windows.''<br /> <br /> * Preparing a "Get Started" tutorial for SIGMOD repeatability <br /> ** Do we have instructions on how to use the latex package without crowdlabs?<br /> *** links to actual workflows in latex<br /> *** have README and example for latex in the source<br /> ** Metadata associated with VisTrails: should we have the ability to add vistrail-level metadata? e.g., who created the vistrail, its purpose, etc.<br /> *** have pointers to paper to workflow and vistrails<br /> *** add pointers to papers<br /> *** allow access to add/edit annotations at the vistrail level<br /> *** ''add GUI element to allow people to edit/add annotations''<br /> <br /> * Maintaining VTK package<br /> ** have four classes that don't wrap, have error even in python level<br /> ** works well with VisTrails otherwise<br /> ** haven't checked the new changes from the wrapping<br /> ** ''Wendel will check on the new version''<br /> <br /> * [[Web services package]] (Tommy)<br /> <br /> === Oct 5, 2010 ===<br /> <br /> * Welcome Tommy!<br /> <br /> * Update on subworkflow (Daniel)<br /> <br /> * Issues raised by Matthias:<br /> <br /> - need to automatically upgrade subworkflows. Currently it is necessary to manually upgrade a subworkflow whenever the version of one of the modules inside them changes.<br /> <br /> - explorations do not detect an MplFigureCell embedded in a subworkflow.</div>

Mates https://www.vistrails.org//index.php?title=Development&diff=3744 Development 2010-12-07T13:19:40Z

<p>Mates: /* Dec 7, 2010 */</p> <hr /> <div>== Roadmap ==<br /> <br /> <br /> == Weekly Meetings ==<br /> <br /> === Dec 7, 2010 ===<br /> <br /> * Question: do we display 'warnings' or only errors?<br /> * Question: Should v1_0_1 MySql schemas be upgraded to v1_0_2?<br /> <br /> * Some suggestions/requests by Matthias:<br /> ** Support the grouping of vistrails into projects<br /> ** Ability to set global parameters/bookmarks=workflows (e.g., a filename or path)<br /> ** Archive results -- create a big tar/zip file with all vt and results<br /> ** Ability to add a vt: if a node is deleted, all data associated with that should also be deleted<br /> ** Create a test suite: run a workflow, save and archive results (user could specify which results should be saved)<br /> <br /> * Data/provenance browser<br /> ** We need a better mechanism that allows users to search/query for workflows and provenance, for example search by project, parameter names and values <br /> ** One way to view this is to allow the creation of mappings between the vistrails/projects into different structures---like "smart folders", e.g., /projectN/vt1/wf3/exec23/O1....<br /> ** Support editing and synchronization in both ways (if edit smart folder, should update vt)<br /> ** Should be able to script the use of the browser, e.g., to test a new version of a workflow against a saved test suite<br /> <br /> * On merging master and crowdlabs git branches<br /> ** some updates on the crowdlabs branch need to be merged/patched into the master branch<br /> ** how should this be done?<br /> <br /> * PythonSource editor<br /> ** modal or multiple window editors?<br /> <br /> * Unit tests<br /> ** [ES]: I ran the test suite on current master branch and 207 tests are executed of which 11 fail. Some of the tests are failing because console_mode_test package can be imported anymore (this is a package created specific to run console_mode tests).<br /> ** Most of the other problems are issues with<br /> <br /> InvalidPipeline: Pipeline cannot be instantiated:<br /> Missing version 0 of package <package><br /> <br /> * Data publishing with CrowdLabs<br /> ** Using git to manage data between local and server data stores might be too messy.<br /> *** Does one create a new git repository for each vistrail on crowdlabs?<br /> *** The local persistent git repo holds all persistent data used on that machine. How does one push/publish just one data product using git?<br /> *** Can anybody think of a plausible git workflow to do this?<br /> ** Perhaps we should just look into adding directory support for RepoSync module along with a better data management UI.<br /> ** Are there OS independent file transfer protocols better than HTTP? Something like rsync might work nicely, but windows support is limited.<br /> <br /> === Nov 30, 2010 ===<br /> <br /> * Subworkflow Upgrades (Daniel)<br /> ** Should be completed finished tonight <br /> ** Local subworkflow upgrades should be working now<br /> ** Need to test these after the checkin<br /> ** Using a dummy controller<br /> <br /> * Assistant for control package (Daniel)<br /> ** Look at examples<br /> ** Try to make an assistant to modify workflow using currently selected modules as looping group<br /> <br /> * [[Improve error handling/reporting]]<br /> ** Tons of message boxes pop up when a workflow has a bunch of errors, UI response is bad after this...<br /> ** Update existing error messages<br /> ** Show Details rendering on Mac<br /> ** What to do when multiple errors occur in a single event?<br /> ** New dialog that shows "next message" but allows user to dismiss messages or view all in the messages window<br /> ** Color coding for messages: use gui/theme.py to configure: maybe try black for warning, gray for log level messages<br /> <br /> * Data and workflow browser (Nivan will give a status update)<br /> <br /> * Saving Explorations (Feature Request from ALPS)<br /> ** We already save parameter explorations with the workflow in the version tree. As is, we save only the last exploration that was done for a given workflow. If you ship the vistrail, the saved parameter explorations will be there. Is this sufficient?<br /> ** [MT] Yes, that takes us a long way. Could one add a button in the explorations to save a certain set of explorations as a new version, like you do with the camera angle in the spreadsheet? That way one could store more than one set of exploration parameters without forcing a new version to be created with every change?<br /> ** Eventually we should provide a selector in parameter exploration<br /> ** Fastest to just allow a null action new version (a bit clunky) but wouldn't require too many changes<br /> <br /> * Clean up/prune persistent directory<br /> ** What are the concrete use cases for this?<br /> ** [MT] Here are two concrete use cases that appeared over the last week:<br /> *** a project is done, the important results copied to an archive and I want to get rid of all its intermediate files, etc., to save space - but not those files associated with other projects. Or, similarly, I'm done testing some workflows, want to remove all (partially buggy or irrelevant) files, and then start with production runs.<br /> *** there was a problem in a workflow(e.g. a bug in HTTPFile module with binary files or a bug in some user code). I want to force a certain persisted directory to be "forgotten", so that I can repair the wrong files. Bumping the version number of the ALPS package would be one way, but that is overkill since it would immediately invalidate all persisted files. Removing the whole persistence directory is just as radical.<br /> ** just testing simple tutorials with small files leaves me with 2 GB of persistent data after 10 days, in 142 persistent directories or files. I need some way to clean this up that is less radical then removing all persistent directories and files. This could be done, by e.g. removing all those which do not occur in tagged or leaf workflows in a certain set of vistrails. Or marking those which occur in certain vistrails with certain labels, and then being able to prune all those with or without some label/tag. Another option might be having the ability to choose from various repositories.<br /> ** [DK] follow up on this<br /> <br /> * Improvements to VisTrails server communication with clients to better handle errors/crashes (Phillip)<br /> ** work in progress<br /> <br /> * Data Publishing with crowdLabs. (Phillip) <br /> ** Do we want data publishing capabilities in VisTrails and crowdLabs? '''YES!!!'''<br /> ** If so...<br /> *** Currently limited data provenance for data. HTTPFile module lacks meta data, RepoSync module has poor usability.<br /> *** Persistent Data works for experimental data. Can we use it for publishing data?<br /> **** Publishing data or storing data on the server and pushing back and forth?<br /> **** What is the current scheme for identifying data on the server?<br /> **** Do we need to do better than the ids, versions, hashes that the peristence package provides?<br /> *** Do we want to utilize data publishing software like [http://thedata.org/home The Dataverse Network] or maybe [http://www.fedora-commons.org/ Fedora Commons]?<br /> ** [ES] Feature request: Can RepoSync support a directory of files?<br /> *** [PM] It can't, and I'm not sure how add it. I also don't really like the current RepoSync interface. It's not obvious how your data is being handled, hence me bring up data publishing, so hopefully we can come up with something more usable/robust.<br /> ** persistence used more for exploratory<br /> ** want to focus on more archive<br /> ** move from exploratory data to published<br /> ** Dataverse: essentially a site to curate data, archive data, host their own site, each university has their own server, linked together<br /> ** persistent store integration: maybe use git to do this, can add to local repositories to move to git<br /> ** use some of the dataverse: standard hash schemes, etc.<br /> ** privacy: can users push workflows without exposing data<br /> ** permissions for who can check things out: download permissions, upload permissions<br /> ** look into setting up persistent store on <br /> <br /> * DisplayWall (Wendel)<br /> ** new version of vistrails receives rotate/scale messages much more slowly<br /> ** using the vtk version that is already there<br /> ** comparing VisTrails 1.2 with nightly src<br /> ** test when the messages are received on the clients<br /> ** [ES] Changed something in the VTKCell, using the Qt interactor from vtk now, just uncomment this (vtkcell.py file)<br /> ** some commands are missed; check why this is happening<br /> ** preferences switch? 2x2 not 2x3<br /> ** check what modifications exist in the client code that's not exactly 1.2?<br /> ** new version of vtk package changed some things<br /> ** update vtk upgrade: resetcamera and addrgbpoint => _# versions<br /> ** Demos on displaywall: Wendel has to keep both VisTrails running and stop work when visitors stop by unannounced<br /> <br /> * Relative paths in workflows, how to deal with them? (Emanuele)<br /> ** [ES] Updated all examples to use HTTPFile for these<br /> ** some of these had to change completely (color widget), kept tree the same format<br /> <br /> === Nov 16, 2010 ===<br /> <br /> * Data and workflow browser (Nivan)<br /> ** Check into git branch (Dave)<br /> ** Add data support to the vistrails, workflows, and execs already exposed in the browser<br /> ** Add thumbnails to the browser<br /> ** Bittorrent support?<br /> <br /> * Upgrade issues -- a wrap? (Daniel)<br /> ** Fixing some issues with namespaces<br /> ** Upgrades for subworkflows that are included with packages, need to save these upgrades with the vt file as a "local abstraction" to ensure provenance. May be able to tie these back to the package versions once the package is upgraded.<br /> <br /> * Test Suite<br /> ** Matthias has his own test suite now<br /> ** We should make sure that our tests are up-to-date and run them before releases<br /> <br /> * Web services on users' guide (Tommy)<br /> ** Users' guide needs only a few changes. One of the changes should be a note making it explicit that we are using a new library and the old one was deprecated.<br /> ** Do we want to update the 39-page guide as well? For now, just update the users' guide. Don't distribute this guide. Just include it in a zip file that includes the old version.<br /> <br /> * [[Improve error handling/reporting]] (Tommy)<br /> ** print statements, has a new window<br /> ** add a menu item to view message window and also keep all messages from the session<br /> ** look into writing QMessageBox errors and warning to message window<br /> ** write all messages to a single window, write critical messages in red?<br /> ** have a utility method that encapsulates the gui error display and the core.debug display (if not running gui, only write to core.debug)<br /> ** don't display stack trace in a qmessagebox? -- try to emulate the invalid pipeline with "Show Details" button.<br /> ** make it easier for users to report errors by having a "copy" or "send to developers" button to report the details of an error.<br /> <br /> * DisplayWall (Wendel)<br /> ** What examples should be in the SVN? If we decide to keep some examples in the svn, some of them use data that should be along.<br /> *** Indicate progress of HTTPFile<br /> *** Had to upgrade examples to work with new examples of VisTrails<br /> *** VisTrails can send progress information with another thread like crowdlabs can do<br /> *** Each machine has a .vistrails and needs to download own version of data, make a single .vistrails for all machines?<br /> *** All slave machines can use the server as a cache?<br /> ** DisplayWall Client on the Apple Store? I dont like the idea of sharing the client source code. And putting the app without login/password will allow anyone to access our displaywall.<br /> *** Add a password for this<br /> *** Do user management?<br /> <br /> * VTK package:<br /> ** Have two ResetCamera modules now<br /> ** Need to write upgrade code<br /> ** Only with vtk 5.7 right now<br /> ** Encode library version in the package version?<br /> ** For reproducibility, we probably want to stick to specific versions<br /> ** We still need some ability to run new versions of vtk for testing, new features, etc., too...<br /> <br /> * Relative paths in workflows, how to deal with them? (Emanuele)<br /> ** Should we use dataDirectory?<br /> ** Change examples to not use hard-coded files?<br /> ** Problem with workflows, not data files<br /> ** On Windows, use ../examples<br /> ** On Mac, use ../../../examples<br /> ** dataDirectory not used, I think<br /> ** create "$VT_EXAMPLES"?<br /> ** problem is using new versions of VisTrails, data may change so referencing old location will fail<br /> ** don't allow users to change this variable<br /> ** convert to use HTTPFile<br /> <br /> * Data files and the GIT Repository (Emanuele)<br /> ** make a separate repository or use the submodule git functionality<br /> ** go with submodule<br /> <br /> * Next release: What still needs to be done?<br /> ** high priority from ALPS list<br /> ** Dan's fixes to the upgrade workflow<br /> ** Date for release? Official release in beginning of December for ALPS<br /> ** All developers should run test suite on push<br /> ** Could make this somewhat automatic, run with nightly release script and email -dev list<br /> ** Could have different classes of tests, make sure that critical ones pass<br /> ** Tests have a bunch of errors currently<br /> <br /> * Saving explorations:<br /> ** Keeps track of the latest exploration for each version<br /> ** Check with Matthias what the requirements for this are<br /> ** Have way to send just specification to a server (run this workflow with this parameter exploration)?<br /> ** Can we have an xml spec that specifies the parameter exploration (export parameter exploration)<br /> ** Currently persisting via xml and then converting to string so we already have an xml serialization<br /> ** Dan wrote API that takes xml string and populates the param_exp gui with the appropriate values<br /> ** Need to back the GUI state with core state to enable short-circuiting the gui<br /> <br /> <br /> * Box input ports should be colored (e.g. black) if they are already set by an internal method. Another color (e.g. red) could be used to indicate mandatory input ports.<br /> ** Currently, non-optional ports are square, shown. Optionals are circles and must be enabled<br /> ** Add a third category for mandatory, problem is backward compat.<br /> ** [ES] Colors are already overloaded, not for ports, use shading instead?<br /> <br /> === Nov 9, 2010 ===<br /> <br /> * FAQ<br /> ** As we reply to users queries, let's add the question and answer to the FAQ!<br /> ** Just a reminder<br /> <br /> * Caching of File module<br /> ** There was a message from a user who got confused because this module is cached by default (and silently). And in his application, since the file actually changed in between runs, he did not see the 'expected' result.<br /> ** Should we keep a hash for files and check whether they changed?<br /> ** Should we no cache File by default?<br /> ** [DK] File is supposed to have a special signature computation that detects changes in the file contents so this may be a bug instead. It does work, it seems to be an issue with the create_file<br /> ** [ES] In this particular case, the file was empty. He deleted the file outside VisTrails and executed it again. As he set create_file to True, he was expecting the file to be created again.<br /> ** [DK] Add random salt to hash signature when the file doesn't exist and create_file is True<br /> ** [JF: DONE] Juliana will edit the FAQ to note that the signatures are dependent on modification times<br /> <br /> * Claurissa will demo different ways to visualize the version tree<br /> ** weighting on various criteria for displaying nodes (importance from session, user, tags, etc.)<br /> ** other modes for viewing the versions (lists or timelines)<br /> ** often see only linear trees, teaching or thinking other than tree?<br /> <br /> <br /> <br /> * Upgrade issues (Dan)<br /> ** Vanilla upgrades should be in today's commit<br /> ** Added code so when doing upgrade to check whether the latest code is upgraded<br /> ** Two types of upgrades: (1) User changes the subworkflow, (2) Modules are out-of-date and need to be upgraded.<br /> ** Add higher-level features like "Upgrade All" for (1), "Import Changes to My Subworkflows", etc.<br /> ** Want a 1-1 mapping between unique subworkflow id and a subworkflow file (even if we copy the same subworkflow to different vistrails)<br /> <br /> * Offscreen bug (update from Huy)<br /> ** Mac issue when switching to Cocoa<br /> ** Huy added a QWidget for each of the render windows that is created, so should be fixed.<br /> <br /> * Gesture support<br /> ** Someone on Windows check the pinch gesture support; it doesn't work for Matthias under Parallels<br /> ** VTKCell is working now<br /> ** Cell dependent, panning now maps to middle button, pinch on the base widget<br /> ** What are Matthias's requirements here? Does touch gestures in the cells matter or only in the vt/wf views<br /> <br /> * Thumbnails: [JF: this needs to go to trac, for 2.0?] [ES: 2 tickets were created for 2.0]<br /> ** Resolution? -- [ES] problem is the size of the vistrail, but this can be configured<br /> ** Can we export to PDF instead of PNG? [ES] cannot display the thumbnail as a PDF in Qt?<br /> ** Can we have the option to save higher-res or PDF versions of the thumbnails<br /> ** Can we have preferences that allow users to save a higher-res or PDF version, maybe a checkbox<br /> ** Maybe associate a high-res version with a spreadsheet cell so that we can save a high-res version on demand<br /> ** high-res version associated with a version but allow user to initiate action from spreadsheet<br /> ** can we have thumbnails dependent on camera position [ES] doesn't like, user won't see exactly that image upon re-executing<br /> ** compact vistrail option by removing thumbnails<br /> <br /> * Tommy has rewritten the Web services module as well as updated all of the examples<br /> ** Server for one of the examples seems to be down<br /> ** Need to update the manual<br /> <br /> * [[Improve error handling/reporting]] (Tommy)<br /> ** Maybe display the intialization messages with the splash screen?<br /> ** Debugging levels have no way to access debug level<br /> ** Tommy takes first cut at trying to determine how to map print statements, notes those he is unsure of<br /> ** Developer guidelines for debug usage, use them instead of print statements in the futur<br /> <br /> * Merge functionality (Tommy) [JF: Cool!]<br /> ** Can now merge an existing vistrail into your own vistrail using menu item<br /> <br /> * Improvements to VisTrails server communication with clients to better handle errors/crashes (Phillip)<br /> ** [PM] I haven't had a chance to implement anything yet.<br /> <br /> * Data Publishing with crowdLabs. (Phillip) <br /> ** Do we want data publishing capabilities in VisTrails and crowdLabs? '''YES!!!'''<br /> ** If so...<br /> *** Currently limited data provenance for data. HTTPFile module lacks meta data, RepoSync module has poor usability.<br /> *** Persistent Data works for experimental data. Can we use it for publishing data?<br /> **** Publishing data or storing data on the server and pushing back and forth?<br /> **** What is the current scheme for identifying data on the server?<br /> **** Do we need to do better than the ids, versions, hashes that the peristence package provides?<br /> *** Do we want to utilize data publishing software like [http://thedata.org/home The Dataverse Network] or maybe [http://www.fedora-commons.org/ Fedora Commons]?<br /> ** [ES] Feature request: Can RepoSync support a directory of files?<br /> *** [PM] It can't, and I'm not sure how add it. I also don't really like the current RepoSync interface. It's not obvious how your data is being handled, hence me bring up data publishing, so hopefully we can come up with something more usable/robust.<br /> <br /> === Nov 3, 2010 ===<br /> <br /> * Upgrading subworkflows (Dan)<br /> ** add updated versions to subworkflow vistrail and push to registry<br /> ** fixed bug with version/descriptor redirects<br /> ** key remaining task is to replace the box representing the subworkflow in the top-level workflow to reflect the change to the upgraded version of the subworkflow<br /> ** many corner cases, but we should get the vanilla upgrade (working) to git<br /> ** need to ensure that each subworkflow file has its own uuid; if a subworkflow exists in two different vistrails, it should have a different uuid (namespace).<br /> ** otherwise, we can get crosstalk where one vt can update the subworkflow from another vt (we can propgate changes to other vts via merges, but this should be a user choice...)<br /> ** enhancements:<br /> *** allow users to merge subworkflows from a file into their own subworkflow (to incorporate outside changes)<br /> *** latest version is the most recent non-upgraded version; if that version has an upgrade, use the upgrade<br /> ***'''Next step: work on usability for controlflow package; explore Dave's idea of an assistant'''<br /> <br /> * Offscreen bug<br /> ** Huy is looking into this<br /> <br /> * Crash due to issues in loaded user packages (Tommy has fixed this)<br /> <br /> * Web services (Tommy)<br /> ** Package is completed, and all but one of our examples are working<br /> ** '''Tommy will test if the Web services will work when there is a proxy'''<br /> <br /> * Improve error handling/reporting (Tommy)<br /> ** we should have a single point for all error messages to pass through that is linked to both core.debug and GUI elements that display error messages. <br /> ** the goal is to implement a function that will be the single point for error messages, and that work without emitting GUI signals in core. As Huy suggested, we should have a GUI-wrapper which will be a no-op when the GUI is not instantiated---this will get rid of some of the pyqt dependencies (at least for the error messages).<br /> <br /> * Improvements to VisTrails server communication with clients to better handle errors/crashes (Phillip)<br /> ** Currently the VisTrails servers consists of: a single-threaded instance with GUI; multiple GUI-less threads<br /> ** We will extend the server API to allow clients to check the server status. The server will provide a separate socket and the client will be able to 'ping' the server; if the server is working properly, it will respond; if it does not respond, the client will have a time out<br /> ** The API will also support the ability to kill both the GUI-less threads and the single-threaded GUI instance. We can then restart the server components using Emanuele's new script.<br /> ** We should also allow users to set a per-workflow time-out; if the workflow execution takes longer than the pre-defined threshold, VisTrails will abort the execution<br /> <br /> <br /> * Update on fixing VTK Package (Wendel)<br /> <br /> ** It seems that the ParaView package has the same issues.<br /> ** '''Wendel has already checked in the new wrapper into the trunk; Emanuele will test it'''<br /> <br /> * Synchronizing VisTrails and ALPS releases (Emanuele)<br /> ** VisTrails latex package was extended so that python is no longer required. If python is not present in the system, instead of issuing a request to execute the workflow and retrieve the image from the Web, a local, previously saved image will be used. It is now also possible to embed the images of workflows into the latex file. Documentation about these and other features are currently in the README file and examples (example.tex) provided with the latex package.<br /> ** Windows and Mac beta binaries already include ALPS<br /> ** Are we distributing ALPS as part of VisTrails then? This seems backwards. ALPS should VisTrails as part of their stuff, no?<br /> *** '''We will have separate binary distributions: one with and one without ALPS'''<br /> <br /> * Is a multi-touch interface to VisTrails possible now that it's supported by Qt? (Phillip)<br /> ** Yes, Qt 4.6+ supports multi-tocuh --- '''Huy will look into this; the goal is to make this work both for the workflow builder/tree view and the spreadsheet'''<br /> ** We still need to figure out what makes sense for multi-touch, however, do we have requirements from ETH-Zurich?<br /> *** We want to map the zoom, pan, click, etc to multi-touch, so that VisTrails can be intractable on a multi-touch screen without a mouse<br /> <br /> * What is the status of the Vismashup i{Phone|Pad|Touch} app? (Phillip)<br /> ** If you mean mac binary, there's an alpha version here: http://www.sci.utah.edu/~emanuele/files/vismashup <br /> ** Do you mean iphone app? [Phillip: yes]<br /> ** '''Wendel will look into this'''<br /> *** One of the issues is how to effectively handle images that are larger than the memory on the iPods and iPads---we need to process these on the server or use a model that allows the image to be manipulated on the client<br /> *** We also need to connect the app with the crowdlabs server, so that the app can get the list of mashups<br /> *** Phillip will investigate the feasibility of working directly with JavaScript and bypass flash<br /> <br /> <br /> <br /> === Oct 19, 2010 ===<br /> <br /> * Update on Trac and Roadmap (Emanuele and David)<br /> ** Trac is now linked to git repository<br /> ** Roadmap on Trac has been cleaned up so we can hide completed milestones<br /> ** Tickets are being reassigned/revisited. Many tickets (24) are not associated with a milestone yet https://vistrails.sci.utah.edu/report/3<br /> * Synchronizing VisTrails and ALPS releases<br /> * Web services support (Tommy)<br /> ** discuss the interaction between the Web Services package and upgrades<br /> * Subworkflows update (Daniel)<br /> * Testing of new module drawing (Erik)<br /> * Aliases<br /> ** Currently, an alias is stored on a parameter. We need aliases to be stored at a higher level so that changing an alias is not a change to a parameter. The one issue is that there is a link between parmaeters and aliases in that an alias can only exist for versions that have the specified parameter. We might also have two versions where the alias points to different parameters. We could just store aliases as a root-level workflow element so that the set of aliases is versioned corrected.<br /> ** In the current implementation, the aliases parsed from the parameters are stored in a dictionary in the workflow, so storing aliases as a root-level workflows element might be the way to go --[[User:Emanuele|Emanuele]] 19:50, 15 October 2010 (MDT)<br /> * Error logging<br /> ** Suggest that we have a single point for all error messages to pass through that is linked to both core.debug and GUI elements that display error messages. This should improve our error handling significantly<br /> * PyQt dependencies in core<br /> ** Can we get rid of PyQt dependencies in core (and db)? This would probably require moving the signals/slots that we currently use to a similar Python implementation which shouldn't be too difficult. This would also require the specification of configuration widgets not as classes but rather some text that can be used to import the GUI elements only when using the GUI code. However, what would happen with workflows that run in command-line mode but require some graphical output?<br /> * Ports<br /> ** One annoying part of the current VisTrails model is that users are not given any visual indication that a port value is already set by a function or vice versa by a connection. In addition, we have no way to specify or enforce cardinality on ports at design time. It seems like we could allow developers to specify when a port should not be connected to more than one value (via a function or a connection), and give some visual feedback when a port has already been specified in one manner.<br /> <br /> ==== Meeting Notes ====<br /> <br /> * Bug regarding userpackages at initial vistrails startup causing startup to fail.<br /> ** Still an issue, ticket still open as a 1.6 milestone<br /> * Copy-paste bug<br /> ** Shortcut not working on Mac. Focus makes shortcut try to copy pipeline.<br /> ** Current fix appends to clipboard as users type - this is not a good fix.<br /> * Synchronizing ALPS and VisTrails releases<br /> ** 1.6 scheduled for Dec. 1<br /> * Webservices<br /> ** New library being used, but it's very low level.<br /> ** Simple types may cause problems with some web-services using complex or XML-based types.<br /> ** Need to make sure that this library can handle at least MOST of the web-services out there.<br /> * Sub-workflows<br /> ** Daniel's changes seem functional and very slick.<br /> ** A little more work on it seems necessary.<br /> ** Need some easy GUI mechanism to delete a subworkflow.<br /> ** Is the exclamation point in the module draw the best way to handle this? Make a tool-tip to explain it.<br /> * VTK Wrapping<br /> ** Changes to VTK Python wrapping is causing headaches - particularly in backwards compatibility.<br /> ** Method typing has changed to expect things like "List" or "Vector" - makes things hard on the user when defining these inputs.<br /> ** Removing List and Vector types and replacing them with dynamically generated versions from Tuples.<br /> * Aliases - Agenda pushed until after Vis.<br /> * Error Logging - Unify some error handling to improve how exceptions are dealt with. More on this after Vis.<br /> * Refactoring out PyQt dependencies from Core/Db/etc - Need to move all the signals/slots into GUI elements. Questions come when a workflow uses GUI elements as inputs. - More after vis.<br /> * Ports - We need a way of showing how a port is specified - function vs. input port. Cardinality must be established in these cases. Ordering on multiple connections should be handled in some way - whether it's just a documentation issue or a more fundamental one.<br /> ** Should we have a tutorial mode that bring up tooltips when someone does something new as if it were a live tutorial.<br /> <br /> <br /> === Oct 12, 2010 ===<br /> <br /> * Trac and Roadmap (Emanuele): Emanuele suggested we should create a development roadmap and make it a permanent item on the agenda. The idea is to go through the Trac tickets and use them to build the roadmap that would be made ''public''.<br /> ** go through trac and sort by priority<br /> ** add other suggestions to roadmap as well<br /> ** admins can add milestones via the admin tab<br /> ** ''David will assign tasks on list; those assigned need to follow up and check and fix them''<br /> <br /> * Update on [[PythonSource error reporting and logging]] (Tommy)<br /> ** ''Need to make sure that all messages are printed through debug.X, so that the debug level (verboseness in preferences) is correctly used''<br /> ** Can cut/copy/paste now<br /> ** Can we prevent the delete from happening?<br /> ** Phillip noted the spyder project has an interactive python console: http://packages.python.org/spyder/<br /> ** Can press enter to get to the end<br /> ** Need to check if this works or if we need to change<br /> ** Now have ability to see stack trace from the GUI (triangle menu)<br /> ** ''Suggest using a dialog to display trace when this item is selected instead of printing to console''<br /> ** Also saving stack trace to the execution log<br /> ** Also printing debug information to vistrails log<br /> ** ''Add trac item to fix prints to use debug''<br /> ** ''Try to fix core.debug to have gui.debug to eliminate Qt dependency''<br /> <br /> * Update on subworkflow (Daniel)<br /> ** Notification when subworkflow is outdated triggered<br /> ** new_abstraction signal from registry in addition to new_module signal so that we can check if we need subworkflows to update<br /> ** How to display the possible upgrades (can be upgraded, can be upgraded but may break, etc.)<br /> <br /> * New utility added by Carlos (currently at scripts/module_appearance) to draw fringes visually, which generates output that can be pasted directly into the add_module call. <br /> ** ''Erik will test this on Mac and Windows.''<br /> <br /> * Preparing a "Get Started" tutorial for SIGMOD repeatability <br /> ** Do we have instructions on how to use the latex package without crowdlabs?<br /> *** links to actual workflows in latex<br /> *** have README and example for latex in the source<br /> ** Metadata associated with VisTrails: should we have the ability to add vistrail-level metadata? e.g., who created the vistrail, its purpose, etc.<br /> *** have pointers to paper to workflow and vistrails<br /> *** add pointers to papers<br /> *** allow access to add/edit annotations at the vistrail level<br /> *** ''add GUI element to allow people to edit/add annotations''<br /> <br /> * Maintaining VTK package<br /> ** have four classes that don't wrap, have error even in python level<br /> ** works well with VisTrails otherwise<br /> ** haven't checked the new changes from the wrapping<br /> ** ''Wendel will check on the new version''<br /> <br /> * [[Web services package]] (Tommy)<br /> <br /> === Oct 5, 2010 ===<br /> <br /> * Welcome Tommy!<br /> <br /> * Update on subworkflow (Daniel)<br /> <br /> * Issues raised by Matthias:<br /> <br /> - need to automatically upgrade subworkflows. Currently it is necessary to manually upgrade a subworkflow whenever the version of one of the modules inside them changes.<br /> <br /> - explorations do not detect an MplFigureCell embedded in a subworkflow.</div>

Mates https://www.vistrails.org//index.php?title=Development&diff=3743 Development 2010-12-07T13:09:08Z

<p>Mates: /* Dec 7, 2010 */</p> <hr /> <div>== Roadmap ==<br /> <br /> <br /> == Weekly Meetings ==<br /> <br /> === Dec 7, 2010 ===<br /> <br /> * Question: do we display 'warnings' or only errors?<br /> * Question: Should v1_0_1 MySql schemas be upgraded to v1_0_2?<br /> <br /> * Some suggestions/requests by Matthias:<br /> ** Support the grouping of vistrails into projects<br /> ** Ability to set global parameters/bookmarks=workflows (e.g., a filename or path)<br /> ** Archive results -- create a big tar/zip file with all vt and results<br /> ** Ability to add a vt: if a node is deleted, all data associated with that should also be deleted<br /> ** Create a test suite: run a workflow, save and archive results (user could specify which results should be saved)<br /> <br /> * Data/provenance browser<br /> ** We need a better mechanism that allows users to search/query for workflows and provenance, for example search by project, parameter names and values <br /> ** One way to view this is to allow the creation of mappings between the vistrails/projects into different structures---like "smart folders", e.g., /projectN/vt1/wf3/exec23/O1....<br /> ** Support editing and synchronization in both ways (if edit smart folder, should update vt)<br /> ** Should be able to script the use of the browser, e.g., to test a new version of a workflow against a saved test suite<br /> <br /> <br /> * On merging master and crowdlabs git branches<br /> ** some updates on the crowdlabs branch need to be merged/patched into the master branch<br /> ** how should this be done?<br /> <br /> * PythonSource editor<br /> ** modal or multiple window editors?<br /> <br /> * Unit tests<br /> ** [ES]: I ran the test suite on current master branch and 207 tests are executed of which 11 fail. Some of the tests are failing because console_mode_test package can be imported anymore (this is a package created specific to run console_mode tests).<br /> ** Most of the other problems are issues with<br /> <br /> InvalidPipeline: Pipeline cannot be instantiated:<br /> Missing version 0 of package <package><br /> <br /> === Nov 30, 2010 ===<br /> <br /> * Subworkflow Upgrades (Daniel)<br /> ** Should be completed finished tonight <br /> ** Local subworkflow upgrades should be working now<br /> ** Need to test these after the checkin<br /> ** Using a dummy controller<br /> <br /> * Assistant for control package (Daniel)<br /> ** Look at examples<br /> ** Try to make an assistant to modify workflow using currently selected modules as looping group<br /> <br /> * [[Improve error handling/reporting]]<br /> ** Tons of message boxes pop up when a workflow has a bunch of errors, UI response is bad after this...<br /> ** Update existing error messages<br /> ** Show Details rendering on Mac<br /> ** What to do when multiple errors occur in a single event?<br /> ** New dialog that shows "next message" but allows user to dismiss messages or view all in the messages window<br /> ** Color coding for messages: use gui/theme.py to configure: maybe try black for warning, gray for log level messages<br /> <br /> * Data and workflow browser (Nivan will give a status update)<br /> <br /> * Saving Explorations (Feature Request from ALPS)<br /> ** We already save parameter explorations with the workflow in the version tree. As is, we save only the last exploration that was done for a given workflow. If you ship the vistrail, the saved parameter explorations will be there. Is this sufficient?<br /> ** [MT] Yes, that takes us a long way. Could one add a button in the explorations to save a certain set of explorations as a new version, like you do with the camera angle in the spreadsheet? That way one could store more than one set of exploration parameters without forcing a new version to be created with every change?<br /> ** Eventually we should provide a selector in parameter exploration<br /> ** Fastest to just allow a null action new version (a bit clunky) but wouldn't require too many changes<br /> <br /> * Clean up/prune persistent directory<br /> ** What are the concrete use cases for this?<br /> ** [MT] Here are two concrete use cases that appeared over the last week:<br /> *** a project is done, the important results copied to an archive and I want to get rid of all its intermediate files, etc., to save space - but not those files associated with other projects. Or, similarly, I'm done testing some workflows, want to remove all (partially buggy or irrelevant) files, and then start with production runs.<br /> *** there was a problem in a workflow(e.g. a bug in HTTPFile module with binary files or a bug in some user code). I want to force a certain persisted directory to be "forgotten", so that I can repair the wrong files. Bumping the version number of the ALPS package would be one way, but that is overkill since it would immediately invalidate all persisted files. Removing the whole persistence directory is just as radical.<br /> ** just testing simple tutorials with small files leaves me with 2 GB of persistent data after 10 days, in 142 persistent directories or files. I need some way to clean this up that is less radical then removing all persistent directories and files. This could be done, by e.g. removing all those which do not occur in tagged or leaf workflows in a certain set of vistrails. Or marking those which occur in certain vistrails with certain labels, and then being able to prune all those with or without some label/tag. Another option might be having the ability to choose from various repositories.<br /> ** [DK] follow up on this<br /> <br /> * Improvements to VisTrails server communication with clients to better handle errors/crashes (Phillip)<br /> ** work in progress<br /> <br /> * Data Publishing with crowdLabs. (Phillip) <br /> ** Do we want data publishing capabilities in VisTrails and crowdLabs? '''YES!!!'''<br /> ** If so...<br /> *** Currently limited data provenance for data. HTTPFile module lacks meta data, RepoSync module has poor usability.<br /> *** Persistent Data works for experimental data. Can we use it for publishing data?<br /> **** Publishing data or storing data on the server and pushing back and forth?<br /> **** What is the current scheme for identifying data on the server?<br /> **** Do we need to do better than the ids, versions, hashes that the peristence package provides?<br /> *** Do we want to utilize data publishing software like [http://thedata.org/home The Dataverse Network] or maybe [http://www.fedora-commons.org/ Fedora Commons]?<br /> ** [ES] Feature request: Can RepoSync support a directory of files?<br /> *** [PM] It can't, and I'm not sure how add it. I also don't really like the current RepoSync interface. It's not obvious how your data is being handled, hence me bring up data publishing, so hopefully we can come up with something more usable/robust.<br /> ** persistence used more for exploratory<br /> ** want to focus on more archive<br /> ** move from exploratory data to published<br /> ** Dataverse: essentially a site to curate data, archive data, host their own site, each university has their own server, linked together<br /> ** persistent store integration: maybe use git to do this, can add to local repositories to move to git<br /> ** use some of the dataverse: standard hash schemes, etc.<br /> ** privacy: can users push workflows without exposing data<br /> ** permissions for who can check things out: download permissions, upload permissions<br /> ** look into setting up persistent store on <br /> <br /> * DisplayWall (Wendel)<br /> ** new version of vistrails receives rotate/scale messages much more slowly<br /> ** using the vtk version that is already there<br /> ** comparing VisTrails 1.2 with nightly src<br /> ** test when the messages are received on the clients<br /> ** [ES] Changed something in the VTKCell, using the Qt interactor from vtk now, just uncomment this (vtkcell.py file)<br /> ** some commands are missed; check why this is happening<br /> ** preferences switch? 2x2 not 2x3<br /> ** check what modifications exist in the client code that's not exactly 1.2?<br /> ** new version of vtk package changed some things<br /> ** update vtk upgrade: resetcamera and addrgbpoint => _# versions<br /> ** Demos on displaywall: Wendel has to keep both VisTrails running and stop work when visitors stop by unannounced<br /> <br /> * Relative paths in workflows, how to deal with them? (Emanuele)<br /> ** [ES] Updated all examples to use HTTPFile for these<br /> ** some of these had to change completely (color widget), kept tree the same format<br /> <br /> === Nov 16, 2010 ===<br /> <br /> * Data and workflow browser (Nivan)<br /> ** Check into git branch (Dave)<br /> ** Add data support to the vistrails, workflows, and execs already exposed in the browser<br /> ** Add thumbnails to the browser<br /> ** Bittorrent support?<br /> <br /> * Upgrade issues -- a wrap? (Daniel)<br /> ** Fixing some issues with namespaces<br /> ** Upgrades for subworkflows that are included with packages, need to save these upgrades with the vt file as a "local abstraction" to ensure provenance. May be able to tie these back to the package versions once the package is upgraded.<br /> <br /> * Test Suite<br /> ** Matthias has his own test suite now<br /> ** We should make sure that our tests are up-to-date and run them before releases<br /> <br /> * Web services on users' guide (Tommy)<br /> ** Users' guide needs only a few changes. One of the changes should be a note making it explicit that we are using a new library and the old one was deprecated.<br /> ** Do we want to update the 39-page guide as well? For now, just update the users' guide. Don't distribute this guide. Just include it in a zip file that includes the old version.<br /> <br /> * [[Improve error handling/reporting]] (Tommy)<br /> ** print statements, has a new window<br /> ** add a menu item to view message window and also keep all messages from the session<br /> ** look into writing QMessageBox errors and warning to message window<br /> ** write all messages to a single window, write critical messages in red?<br /> ** have a utility method that encapsulates the gui error display and the core.debug display (if not running gui, only write to core.debug)<br /> ** don't display stack trace in a qmessagebox? -- try to emulate the invalid pipeline with "Show Details" button.<br /> ** make it easier for users to report errors by having a "copy" or "send to developers" button to report the details of an error.<br /> <br /> * DisplayWall (Wendel)<br /> ** What examples should be in the SVN? If we decide to keep some examples in the svn, some of them use data that should be along.<br /> *** Indicate progress of HTTPFile<br /> *** Had to upgrade examples to work with new examples of VisTrails<br /> *** VisTrails can send progress information with another thread like crowdlabs can do<br /> *** Each machine has a .vistrails and needs to download own version of data, make a single .vistrails for all machines?<br /> *** All slave machines can use the server as a cache?<br /> ** DisplayWall Client on the Apple Store? I dont like the idea of sharing the client source code. And putting the app without login/password will allow anyone to access our displaywall.<br /> *** Add a password for this<br /> *** Do user management?<br /> <br /> * VTK package:<br /> ** Have two ResetCamera modules now<br /> ** Need to write upgrade code<br /> ** Only with vtk 5.7 right now<br /> ** Encode library version in the package version?<br /> ** For reproducibility, we probably want to stick to specific versions<br /> ** We still need some ability to run new versions of vtk for testing, new features, etc., too...<br /> <br /> * Relative paths in workflows, how to deal with them? (Emanuele)<br /> ** Should we use dataDirectory?<br /> ** Change examples to not use hard-coded files?<br /> ** Problem with workflows, not data files<br /> ** On Windows, use ../examples<br /> ** On Mac, use ../../../examples<br /> ** dataDirectory not used, I think<br /> ** create "$VT_EXAMPLES"?<br /> ** problem is using new versions of VisTrails, data may change so referencing old location will fail<br /> ** don't allow users to change this variable<br /> ** convert to use HTTPFile<br /> <br /> * Data files and the GIT Repository (Emanuele)<br /> ** make a separate repository or use the submodule git functionality<br /> ** go with submodule<br /> <br /> * Next release: What still needs to be done?<br /> ** high priority from ALPS list<br /> ** Dan's fixes to the upgrade workflow<br /> ** Date for release? Official release in beginning of December for ALPS<br /> ** All developers should run test suite on push<br /> ** Could make this somewhat automatic, run with nightly release script and email -dev list<br /> ** Could have different classes of tests, make sure that critical ones pass<br /> ** Tests have a bunch of errors currently<br /> <br /> * Saving explorations:<br /> ** Keeps track of the latest exploration for each version<br /> ** Check with Matthias what the requirements for this are<br /> ** Have way to send just specification to a server (run this workflow with this parameter exploration)?<br /> ** Can we have an xml spec that specifies the parameter exploration (export parameter exploration)<br /> ** Currently persisting via xml and then converting to string so we already have an xml serialization<br /> ** Dan wrote API that takes xml string and populates the param_exp gui with the appropriate values<br /> ** Need to back the GUI state with core state to enable short-circuiting the gui<br /> <br /> <br /> * Box input ports should be colored (e.g. black) if they are already set by an internal method. Another color (e.g. red) could be used to indicate mandatory input ports.<br /> ** Currently, non-optional ports are square, shown. Optionals are circles and must be enabled<br /> ** Add a third category for mandatory, problem is backward compat.<br /> ** [ES] Colors are already overloaded, not for ports, use shading instead?<br /> <br /> === Nov 9, 2010 ===<br /> <br /> * FAQ<br /> ** As we reply to users queries, let's add the question and answer to the FAQ!<br /> ** Just a reminder<br /> <br /> * Caching of File module<br /> ** There was a message from a user who got confused because this module is cached by default (and silently). And in his application, since the file actually changed in between runs, he did not see the 'expected' result.<br /> ** Should we keep a hash for files and check whether they changed?<br /> ** Should we no cache File by default?<br /> ** [DK] File is supposed to have a special signature computation that detects changes in the file contents so this may be a bug instead. It does work, it seems to be an issue with the create_file<br /> ** [ES] In this particular case, the file was empty. He deleted the file outside VisTrails and executed it again. As he set create_file to True, he was expecting the file to be created again.<br /> ** [DK] Add random salt to hash signature when the file doesn't exist and create_file is True<br /> ** [JF: DONE] Juliana will edit the FAQ to note that the signatures are dependent on modification times<br /> <br /> * Claurissa will demo different ways to visualize the version tree<br /> ** weighting on various criteria for displaying nodes (importance from session, user, tags, etc.)<br /> ** other modes for viewing the versions (lists or timelines)<br /> ** often see only linear trees, teaching or thinking other than tree?<br /> <br /> <br /> <br /> * Upgrade issues (Dan)<br /> ** Vanilla upgrades should be in today's commit<br /> ** Added code so when doing upgrade to check whether the latest code is upgraded<br /> ** Two types of upgrades: (1) User changes the subworkflow, (2) Modules are out-of-date and need to be upgraded.<br /> ** Add higher-level features like "Upgrade All" for (1), "Import Changes to My Subworkflows", etc.<br /> ** Want a 1-1 mapping between unique subworkflow id and a subworkflow file (even if we copy the same subworkflow to different vistrails)<br /> <br /> * Offscreen bug (update from Huy)<br /> ** Mac issue when switching to Cocoa<br /> ** Huy added a QWidget for each of the render windows that is created, so should be fixed.<br /> <br /> * Gesture support<br /> ** Someone on Windows check the pinch gesture support; it doesn't work for Matthias under Parallels<br /> ** VTKCell is working now<br /> ** Cell dependent, panning now maps to middle button, pinch on the base widget<br /> ** What are Matthias's requirements here? Does touch gestures in the cells matter or only in the vt/wf views<br /> <br /> * Thumbnails: [JF: this needs to go to trac, for 2.0?] [ES: 2 tickets were created for 2.0]<br /> ** Resolution? -- [ES] problem is the size of the vistrail, but this can be configured<br /> ** Can we export to PDF instead of PNG? [ES] cannot display the thumbnail as a PDF in Qt?<br /> ** Can we have the option to save higher-res or PDF versions of the thumbnails<br /> ** Can we have preferences that allow users to save a higher-res or PDF version, maybe a checkbox<br /> ** Maybe associate a high-res version with a spreadsheet cell so that we can save a high-res version on demand<br /> ** high-res version associated with a version but allow user to initiate action from spreadsheet<br /> ** can we have thumbnails dependent on camera position [ES] doesn't like, user won't see exactly that image upon re-executing<br /> ** compact vistrail option by removing thumbnails<br /> <br /> * Tommy has rewritten the Web services module as well as updated all of the examples<br /> ** Server for one of the examples seems to be down<br /> ** Need to update the manual<br /> <br /> * [[Improve error handling/reporting]] (Tommy)<br /> ** Maybe display the intialization messages with the splash screen?<br /> ** Debugging levels have no way to access debug level<br /> ** Tommy takes first cut at trying to determine how to map print statements, notes those he is unsure of<br /> ** Developer guidelines for debug usage, use them instead of print statements in the futur<br /> <br /> * Merge functionality (Tommy) [JF: Cool!]<br /> ** Can now merge an existing vistrail into your own vistrail using menu item<br /> <br /> * Improvements to VisTrails server communication with clients to better handle errors/crashes (Phillip)<br /> ** [PM] I haven't had a chance to implement anything yet.<br /> <br /> * Data Publishing with crowdLabs. (Phillip) <br /> ** Do we want data publishing capabilities in VisTrails and crowdLabs? '''YES!!!'''<br /> ** If so...<br /> *** Currently limited data provenance for data. HTTPFile module lacks meta data, RepoSync module has poor usability.<br /> *** Persistent Data works for experimental data. Can we use it for publishing data?<br /> **** Publishing data or storing data on the server and pushing back and forth?<br /> **** What is the current scheme for identifying data on the server?<br /> **** Do we need to do better than the ids, versions, hashes that the peristence package provides?<br /> *** Do we want to utilize data publishing software like [http://thedata.org/home The Dataverse Network] or maybe [http://www.fedora-commons.org/ Fedora Commons]?<br /> ** [ES] Feature request: Can RepoSync support a directory of files?<br /> *** [PM] It can't, and I'm not sure how add it. I also don't really like the current RepoSync interface. It's not obvious how your data is being handled, hence me bring up data publishing, so hopefully we can come up with something more usable/robust.<br /> <br /> === Nov 3, 2010 ===<br /> <br /> * Upgrading subworkflows (Dan)<br /> ** add updated versions to subworkflow vistrail and push to registry<br /> ** fixed bug with version/descriptor redirects<br /> ** key remaining task is to replace the box representing the subworkflow in the top-level workflow to reflect the change to the upgraded version of the subworkflow<br /> ** many corner cases, but we should get the vanilla upgrade (working) to git<br /> ** need to ensure that each subworkflow file has its own uuid; if a subworkflow exists in two different vistrails, it should have a different uuid (namespace).<br /> ** otherwise, we can get crosstalk where one vt can update the subworkflow from another vt (we can propgate changes to other vts via merges, but this should be a user choice...)<br /> ** enhancements:<br /> *** allow users to merge subworkflows from a file into their own subworkflow (to incorporate outside changes)<br /> *** latest version is the most recent non-upgraded version; if that version has an upgrade, use the upgrade<br /> ***'''Next step: work on usability for controlflow package; explore Dave's idea of an assistant'''<br /> <br /> * Offscreen bug<br /> ** Huy is looking into this<br /> <br /> * Crash due to issues in loaded user packages (Tommy has fixed this)<br /> <br /> * Web services (Tommy)<br /> ** Package is completed, and all but one of our examples are working<br /> ** '''Tommy will test if the Web services will work when there is a proxy'''<br /> <br /> * Improve error handling/reporting (Tommy)<br /> ** we should have a single point for all error messages to pass through that is linked to both core.debug and GUI elements that display error messages. <br /> ** the goal is to implement a function that will be the single point for error messages, and that work without emitting GUI signals in core. As Huy suggested, we should have a GUI-wrapper which will be a no-op when the GUI is not instantiated---this will get rid of some of the pyqt dependencies (at least for the error messages).<br /> <br /> * Improvements to VisTrails server communication with clients to better handle errors/crashes (Phillip)<br /> ** Currently the VisTrails servers consists of: a single-threaded instance with GUI; multiple GUI-less threads<br /> ** We will extend the server API to allow clients to check the server status. The server will provide a separate socket and the client will be able to 'ping' the server; if the server is working properly, it will respond; if it does not respond, the client will have a time out<br /> ** The API will also support the ability to kill both the GUI-less threads and the single-threaded GUI instance. We can then restart the server components using Emanuele's new script.<br /> ** We should also allow users to set a per-workflow time-out; if the workflow execution takes longer than the pre-defined threshold, VisTrails will abort the execution<br /> <br /> <br /> * Update on fixing VTK Package (Wendel)<br /> <br /> ** It seems that the ParaView package has the same issues.<br /> ** '''Wendel has already checked in the new wrapper into the trunk; Emanuele will test it'''<br /> <br /> * Synchronizing VisTrails and ALPS releases (Emanuele)<br /> ** VisTrails latex package was extended so that python is no longer required. If python is not present in the system, instead of issuing a request to execute the workflow and retrieve the image from the Web, a local, previously saved image will be used. It is now also possible to embed the images of workflows into the latex file. Documentation about these and other features are currently in the README file and examples (example.tex) provided with the latex package.<br /> ** Windows and Mac beta binaries already include ALPS<br /> ** Are we distributing ALPS as part of VisTrails then? This seems backwards. ALPS should VisTrails as part of their stuff, no?<br /> *** '''We will have separate binary distributions: one with and one without ALPS'''<br /> <br /> * Is a multi-touch interface to VisTrails possible now that it's supported by Qt? (Phillip)<br /> ** Yes, Qt 4.6+ supports multi-tocuh --- '''Huy will look into this; the goal is to make this work both for the workflow builder/tree view and the spreadsheet'''<br /> ** We still need to figure out what makes sense for multi-touch, however, do we have requirements from ETH-Zurich?<br /> *** We want to map the zoom, pan, click, etc to multi-touch, so that VisTrails can be intractable on a multi-touch screen without a mouse<br /> <br /> * What is the status of the Vismashup i{Phone|Pad|Touch} app? (Phillip)<br /> ** If you mean mac binary, there's an alpha version here: http://www.sci.utah.edu/~emanuele/files/vismashup <br /> ** Do you mean iphone app? [Phillip: yes]<br /> ** '''Wendel will look into this'''<br /> *** One of the issues is how to effectively handle images that are larger than the memory on the iPods and iPads---we need to process these on the server or use a model that allows the image to be manipulated on the client<br /> *** We also need to connect the app with the crowdlabs server, so that the app can get the list of mashups<br /> *** Phillip will investigate the feasibility of working directly with JavaScript and bypass flash<br /> <br /> <br /> <br /> === Oct 19, 2010 ===<br /> <br /> * Update on Trac and Roadmap (Emanuele and David)<br /> ** Trac is now linked to git repository<br /> ** Roadmap on Trac has been cleaned up so we can hide completed milestones<br /> ** Tickets are being reassigned/revisited. Many tickets (24) are not associated with a milestone yet https://vistrails.sci.utah.edu/report/3<br /> * Synchronizing VisTrails and ALPS releases<br /> * Web services support (Tommy)<br /> ** discuss the interaction between the Web Services package and upgrades<br /> * Subworkflows update (Daniel)<br /> * Testing of new module drawing (Erik)<br /> * Aliases<br /> ** Currently, an alias is stored on a parameter. We need aliases to be stored at a higher level so that changing an alias is not a change to a parameter. The one issue is that there is a link between parmaeters and aliases in that an alias can only exist for versions that have the specified parameter. We might also have two versions where the alias points to different parameters. We could just store aliases as a root-level workflow element so that the set of aliases is versioned corrected.<br /> ** In the current implementation, the aliases parsed from the parameters are stored in a dictionary in the workflow, so storing aliases as a root-level workflows element might be the way to go --[[User:Emanuele|Emanuele]] 19:50, 15 October 2010 (MDT)<br /> * Error logging<br /> ** Suggest that we have a single point for all error messages to pass through that is linked to both core.debug and GUI elements that display error messages. This should improve our error handling significantly<br /> * PyQt dependencies in core<br /> ** Can we get rid of PyQt dependencies in core (and db)? This would probably require moving the signals/slots that we currently use to a similar Python implementation which shouldn't be too difficult. This would also require the specification of configuration widgets not as classes but rather some text that can be used to import the GUI elements only when using the GUI code. However, what would happen with workflows that run in command-line mode but require some graphical output?<br /> * Ports<br /> ** One annoying part of the current VisTrails model is that users are not given any visual indication that a port value is already set by a function or vice versa by a connection. In addition, we have no way to specify or enforce cardinality on ports at design time. It seems like we could allow developers to specify when a port should not be connected to more than one value (via a function or a connection), and give some visual feedback when a port has already been specified in one manner.<br /> <br /> ==== Meeting Notes ====<br /> <br /> * Bug regarding userpackages at initial vistrails startup causing startup to fail.<br /> ** Still an issue, ticket still open as a 1.6 milestone<br /> * Copy-paste bug<br /> ** Shortcut not working on Mac. Focus makes shortcut try to copy pipeline.<br /> ** Current fix appends to clipboard as users type - this is not a good fix.<br /> * Synchronizing ALPS and VisTrails releases<br /> ** 1.6 scheduled for Dec. 1<br /> * Webservices<br /> ** New library being used, but it's very low level.<br /> ** Simple types may cause problems with some web-services using complex or XML-based types.<br /> ** Need to make sure that this library can handle at least MOST of the web-services out there.<br /> * Sub-workflows<br /> ** Daniel's changes seem functional and very slick.<br /> ** A little more work on it seems necessary.<br /> ** Need some easy GUI mechanism to delete a subworkflow.<br /> ** Is the exclamation point in the module draw the best way to handle this? Make a tool-tip to explain it.<br /> * VTK Wrapping<br /> ** Changes to VTK Python wrapping is causing headaches - particularly in backwards compatibility.<br /> ** Method typing has changed to expect things like "List" or "Vector" - makes things hard on the user when defining these inputs.<br /> ** Removing List and Vector types and replacing them with dynamically generated versions from Tuples.<br /> * Aliases - Agenda pushed until after Vis.<br /> * Error Logging - Unify some error handling to improve how exceptions are dealt with. More on this after Vis.<br /> * Refactoring out PyQt dependencies from Core/Db/etc - Need to move all the signals/slots into GUI elements. Questions come when a workflow uses GUI elements as inputs. - More after vis.<br /> * Ports - We need a way of showing how a port is specified - function vs. input port. Cardinality must be established in these cases. Ordering on multiple connections should be handled in some way - whether it's just a documentation issue or a more fundamental one.<br /> ** Should we have a tutorial mode that bring up tooltips when someone does something new as if it were a live tutorial.<br /> <br /> <br /> === Oct 12, 2010 ===<br /> <br /> * Trac and Roadmap (Emanuele): Emanuele suggested we should create a development roadmap and make it a permanent item on the agenda. The idea is to go through the Trac tickets and use them to build the roadmap that would be made ''public''.<br /> ** go through trac and sort by priority<br /> ** add other suggestions to roadmap as well<br /> ** admins can add milestones via the admin tab<br /> ** ''David will assign tasks on list; those assigned need to follow up and check and fix them''<br /> <br /> * Update on [[PythonSource error reporting and logging]] (Tommy)<br /> ** ''Need to make sure that all messages are printed through debug.X, so that the debug level (verboseness in preferences) is correctly used''<br /> ** Can cut/copy/paste now<br /> ** Can we prevent the delete from happening?<br /> ** Phillip noted the spyder project has an interactive python console: http://packages.python.org/spyder/<br /> ** Can press enter to get to the end<br /> ** Need to check if this works or if we need to change<br /> ** Now have ability to see stack trace from the GUI (triangle menu)<br /> ** ''Suggest using a dialog to display trace when this item is selected instead of printing to console''<br /> ** Also saving stack trace to the execution log<br /> ** Also printing debug information to vistrails log<br /> ** ''Add trac item to fix prints to use debug''<br /> ** ''Try to fix core.debug to have gui.debug to eliminate Qt dependency''<br /> <br /> * Update on subworkflow (Daniel)<br /> ** Notification when subworkflow is outdated triggered<br /> ** new_abstraction signal from registry in addition to new_module signal so that we can check if we need subworkflows to update<br /> ** How to display the possible upgrades (can be upgraded, can be upgraded but may break, etc.)<br /> <br /> * New utility added by Carlos (currently at scripts/module_appearance) to draw fringes visually, which generates output that can be pasted directly into the add_module call. <br /> ** ''Erik will test this on Mac and Windows.''<br /> <br /> * Preparing a "Get Started" tutorial for SIGMOD repeatability <br /> ** Do we have instructions on how to use the latex package without crowdlabs?<br /> *** links to actual workflows in latex<br /> *** have README and example for latex in the source<br /> ** Metadata associated with VisTrails: should we have the ability to add vistrail-level metadata? e.g., who created the vistrail, its purpose, etc.<br /> *** have pointers to paper to workflow and vistrails<br /> *** add pointers to papers<br /> *** allow access to add/edit annotations at the vistrail level<br /> *** ''add GUI element to allow people to edit/add annotations''<br /> <br /> * Maintaining VTK package<br /> ** have four classes that don't wrap, have error even in python level<br /> ** works well with VisTrails otherwise<br /> ** haven't checked the new changes from the wrapping<br /> ** ''Wendel will check on the new version''<br /> <br /> * [[Web services package]] (Tommy)<br /> <br /> === Oct 5, 2010 ===<br /> <br /> * Welcome Tommy!<br /> <br /> * Update on subworkflow (Daniel)<br /> <br /> * Issues raised by Matthias:<br /> <br /> - need to automatically upgrade subworkflows. Currently it is necessary to manually upgrade a subworkflow whenever the version of one of the modules inside them changes.<br /> <br /> - explorations do not detect an MplFigureCell embedded in a subworkflow.</div>

Mates https://www.vistrails.org//index.php?title=FAQ&diff=3735 FAQ 2010-12-06T13:43:13Z

<p>Mates: /* = How do I execute workflows and control VisTrails through another application? */</p> <hr /> <div>Also check our [[known_issues|Known Issues]] page for troubleshooting.<br /> <br /> ----<br /> == Running workflows ==<br /> <br /> === How can I run a workflow using the command line? ===<br /> <br /> ('''Updated for version 1.2''') Call vistrails using the following options:<br /> <br /> <code>python vistrails.py -b path_to_vistrails_file:pipeline </code><br /> <br /> where pipeline can be a version '''tag name''' or version '''id'''<br /> <br /> '''NOTE:''' If you downloaded the MacOS X bundle, you can run vistrails from the command line via the following commands in the Terminal. Change the current directory to wherever VisTrails was installed (often <code>/Applications</code>), and then type:<br /> <br /> <code>Vistrails.app/Contents/MacOS/vistrails [<cmd_line_options>]</code><br /> <br /> === Using the command line, we'd like to execute a workflow multiple times, with slightly different parameters, and create a series of output files. Is this possible?===<br /> <br /> ('''Updated for version 1.2''') We can change parameters that have an alias through the command line.<br /> <br /> For example, offscreen pipeline in offscreen.vt always creates the file called image.png. If you want generate it with a different filename:<br /> <br /> <code>python vistrails.py -b ../examples/offscreen.vt:offscreen -a"filename=other.png"</code><br /> <br /> <code>filename</code> in the example above is the alias name assigned to the parameter in the <code>value</code> method inside the String module. When running a pipeline from the command line, VisTrails will try to start the spreadsheet automatically if the pipeline requires it. For example, this other execution will also start the spreadsheet:<br /> <br /> <code>python vistrails.py -b ../examples/head.vt:aliases \<br /> -a"isovalue=30&&Diffuse_Color_R=0.8&&Diffuse_Color_G=0.4&&Diffuse_Color_B=0.2"</code><br /> <br /> You can also execute more than one pipeline on the command line:<br /> <br /> <code>python vistrails.py -b ../examples/head.vt:aliases ../examples/spx.vt:spx \<br /> -a"isovalue=30"</code><br /> <br /> Use the -a parameter only once regardless the number of pipelines.<br /> <br /> === I can load a vistrail, and the version tree shows up fine. However, no pipelines appear when I click on a version. What gives? ===<br /> <br /> The most likely reason is that the vistrail uses a package that is not registered with VisTrails. You need to identify the needed package and add it to your <code>.vistrails/startup.py</code>. A single line like the following should be enough:<br /> <br /> addPackage('enter_package_name_here')<br /> <br /> Some packages might need more information. For example:<br /> <br /> addPackage('afront', executable_path='/path/to/afront')<br /> <br /> Refer to the package documentation for details. The one inconvenient step is that currently there's no automated way to describe what is the missing package. We're working on this feature for future releases.<br /> <br /> === I have a workflow that reads a file and then does some processing. The first time it runs, it executes correctly. But in subsequent, nothing happens. ===<br /> <br /> VisTrails caches by default, so after a workflow is executed, if none of its parameters change, it won't be executed again. <br /> <br /> If a workflow reads a file using the basic module File, VisTrails does check whether the file was modified since the last run. It does so by keeping a signature that is based on the modification time of the file. And if the file was modified, the File module and all downstream modules (the ones which depend on File) will be executed. <br /> <br /> <br /> ''Note: If you would like your input and output data to be versioned, you can use the Persistence package.''<br /> <br /> If you do not want VisTrails to cache executions, you can turn off caching: go to Menu Edit -> Preferences and in the General Configuration tab, change Cache execution results to Never.<br /> <br /> === Can VisTrails execute workflows in parallel? ===<br /> <br /> The VisTrails server can only execute pipelines in parallel if there's more than one instance of VisTrails running. The command <br /> <br /> <code> self.rpcserver = ThreadedXMLRPCServer((self.temp_xml_rpc_options.server, self.temp_xml_rpc_options.port))</code><br /> <br /> starts a multithreaded version of the XML-RPC server, so it will create a thread for each request received by the server. The problem is that Qt/PyQT doesn't allow these multiple threads create GUI objects, only in the main thread. To overcome this limitation, the multithreaded version can instantiate other single threaded versions of VisTrails and put them in a queue, so workflow executions and other GUI-related requests, such as generating workflow graphs and history trees can be forwarded to this queue, and each instance takes turns in answering the request. If the results are in the cache, the multithreaded version answers the requests directly.<br /> <br /> Note that this infrastructure works on Linux only. To make this work on Windows, you have to create a script similar to start_vistrails_xvfb.sh (located in the scripts folder) where you can send the number of other instances via command-line options to VisTrails. The command line options are:<br /> <br /> python vistrails_server.py -T <ADDRESS> -R <PORT> -O<NUMBER_OF_OTHER_VISTRAILS_INSTANCES> [-M]&<br /> <br /> If you want the main vistrails instance to be multithreaded, use the -M at the end.<br /> <br /> After creating this script, update function start_other_instances in vistrails/gui/application_server.py lines 1007-1023 and set the script variable to point to your script. You may also have to change the arguments sent to your script (line 1016: for example, you don't need to set a virtual display). You will need to change the path to the stop_vistrails_server.py script (on line 1026) according to your installation path.<br /> <br /> === When a workflow is executed, what do the colors mean? ===<br /> <br /> - lilac: module was notexecuted<br /> <br /> - yellow: module is currently being executed <br /> <br /> - green: module was successfully executed <br /> <br /> - orange: module was cached<br /> <br /> - red: the execution of the module failed <br /> <br /> == Building workflows ==<br /> <br /> === Is there a way to give each widget a "display name" in addition to the module name at the center of the widget? ===<br /> <br /> Yes, but it is not easily accessible from the GUI and it definitely needs to be more intuitive. For now, we use the annotation value of key "__desc__" as a module label. If you want to set a PythonSource label, you have to select the module. Then click on the Annotation tab, and add a key named "__desc__", whatever value you set to this key will be the label. We are currently working on a new interface for this functionality.<br /> <br /> === Is there a way to re-center the picture-in-picture (PiP) view? ===<br /> <br /> Yes. If you click on the PIP window to bring it to focus, you can press Ctrl-R (or Command-R on Mac) to re-center the PiP window.<br /> <br /> === How do I search for a literal "?" (question mark) in the search box in the Property panel? ===<br /> <br /> Since we allow regular expressions in our search box, question marks are treated as meta-characters. Thus, searching for "?" returns everything and "abc?" will return everything containing "abc". You need to use "\?" instead to search for "?". So the search for "??" would be "\?\?".<br /> <br /> == Using VisTrails as a server ==<br /> <br /> === What is the VisTrails server-mode? ===<br /> <br /> Using the VisTrails server mode, it is possible to ''execute workflows and control VisTrails through another application''. For example, the CrowdLabs Web portal (http://www.crowdlabs.org) accesses a VisTrails sever to execute workflows, retrieve and display vistrail trees and workflows.<br /> <br /> === How do I execute workflows and control VisTrails through another application? ===<br /> <br /> The way you access the server is by doing XML-RPC calls. In the current VisTrails release, we include a set of PHP scripts that can talk to a VisTrails server instance. They are in "extensions/http" folder. The files are reasonably well documented. Also, it should be not difficult to create python scripts to access the server (just use xmlrpclib module).<br /> <br /> Note that the VisTrails server requires the provenance and workflows to be in a database. More detailed instructions on how to setup the server and the database<br /> are available here:<br /> <br /> http://www.crowdlabs.org/site_media/static/dev_docs/vistrails_server_setup.html<br /> <br /> http://www.crowdlabs.org/site_media/static/dev_docs/vistrails_database_setup.html<br /> <br /> If what you want is just to execute a series of workflows in batch mode, a simpler solution would be to ''use the VisTrails client in batch mode''. Chapter 12 of the user's guide contains detailed information and examples on that.<br /> <br /> == Control Flow ==<br /> <br /> === Note: using map ===<br /> <br /> When using 'map', the module (or subworkflow) used as function port in the map module MUST be a function, i.e., it can only define 1 output port. <br /> <br /> == Spreadsheet ==<br /> <br /> Where ''pipeline'' is a version number or a tag.<br /> <br /> === How can I save an image from the spreadsheet? ===<br /> <br /> While having the focus on a spreadsheet cell and select the camera on the toolbar to take a snapshot. The system will prompt you for the location and file name where it should be saved. The other icons can be used for saving multiple images that can be used for generating an animation on demand. A whole sheet can also be saved by selecting Export (either from the menu or from the toolbar).<br /> <br /> ===Is it possible to save the complete state of the spreadsheet?===<br /> <br /> === Can I view multiple sheets at the same time? ===<br /> <br /> Yes. Each sheet on the spreadsheet can be displayed as a dock widget separated from the main spreadsheet window by dragging its tab name out of the tab bar at the bottom of the spreadsheet.<br /> <br /> === Then, how can I put back a separated sheet? ===<br /> <br /> A sheet can be docked back to the main window by dragging it back to the tab bar or double-click on its title bar.<br /> <br /> === How can I order sheets on the spreadsheet? ===<br /> <br /> This can be done by dragging the sheet name on the bottom top bar and drop it to the right place.<br /> <br /> === Can I control where a cell will be placed on the spreadsheet window? ===<br /> <br /> By default, an unoccupied cell on the active sheet will be chosen to display the result. However, you can specify exactly in the pipeline where a spreadsheet cell will be placed by using CellLocation and SheetReference. CellLocation specifies the location (row and column) of a cell when connecting to a spreadsheet cell (VTKCell, ImageViewerCell, ...). Similarly, a SheetReference module (when connecting to a CellLocation) will specify which sheet the cell will be put on given its name, minimum row size and minimum column size. There is an example of this in examples/vtk.xml (select the version below Double Renderer).<br /> <br /> === How do I output results to the spreadsheet? ===<br /> <br /> By inspecting the VisTrails Spreadsheet package (in the list of packages, to the left of the pipeline builder), you<br /> can see there are built-in cells for different kinds of data, e.g., RichTextCell to display HTML and plain text. op<br /> You (the user) can also define new cell types to display application-specific data. For example, we have developed<br /> VtkCell, MplFigureCell, and OpenGLCell. It is possible to display pretty much anything on the Spreadsheet!<br /> <br /> Examples of writing cell modules can be found in:<br /> RichTextCell: packages/spreadsheet/widgets/richtext/richtext.py<br /> VTK: packages/vtk/vtkcell.py<br /> <br /> Here is the summary of some requirements on a cell widget:<br /> <br /> (1) It must be a Qt widget. It should inherit from spreadsheet_cell.QCellWidget in the spreadsheet package. Although any Qt Widget would work, certain features such as animation will not be available (without rewriting it).<br /> <br /> (2) It must re-implement the updateContents() function to take a set of inputs (usually coming from input ports of a wrapper Module) and display on the cells. VisTrails uses this function to update/reuse cells on the spreadsheet when new data comes in.<br /> <br /> (3) It needs a wrapper VisTrails Module (inherited from basic_widgets.SpreadsheetCell of the spreadsheet package). Inside the compute() method of this module, it may call self.display(CellWidgetType, (inputs)) to trigger the display event on the spreadsheet.<br /> <br /> === How do I control the default number of cells in the spreadsheet? ===<br /> <br /> You can configure the rowCount and colCount using the preferences dialog. Just go to the Module Packages tab, select spreadsheet in the "Enabled packages" and press the Configure button. Then a list of all the configuration options for the spreadsheet will show up.<br /> <br /> === Is it possible to launch a web browser from the vistrails spreadsheet? We would like to output several urls from a parameter sweep and then have the option to click on each one to view the resulting page. I can view the page within the spreadsheet, but it is really too crowded.===<br /> <br /> Currently, there isn't a widget that provides exactly this functionality, but I can think of a few solutions that may work for you:<br /> <br /> (1) You can use parameter exploration to generate multiple sheets so you might have an exploration that opens each page in a new sheet. Use the third column/dimension in the exploration interface to have a parameter span sheets.<br /> <br /> (2) The spreadsheet is extensible so you can write a custom spreadsheet cell widget that has a button or label with the desired link (a QLabel with openExternalLinks set to True, for example).<br /> <br /> (3) You can tweak the existing RichTextCell be adding the line "self.browser.setOpenExternalLinks(True)" at line 63 of the source file "vistrails/packages/spreadsheet/widgets/richtext/richtext.py". Then, if your workflow creates a file with html markup text like "<a href="http://www.vistrails.org">VisTrails</a>" connected to a RichTextCell, clicking on the rendered link in the cell will open it in a web browser. You need to add the aforementioned line to the source to let Qt know that you want the link opened externally; by default, it will just issue an event that isn't processed.<br /> <br /> == Integrating your software into VisTrails ==<br /> <br /> ===How can I integrate my own program into VisTrails?===<br /> <br /> The easiest way is to create a package. Writing a package is often very simple, here are instructions on how to do it: [[UsersGuideVisTrailsPackages]]<br /> <br /> You can also dynamically generate modules. For an example see:<br /> <br /> http://www.vistrails.org/index.php/UsersGuideVisTrailsPackages#Packages_that_generate_modules_dynamically<br /> <br /> In particular, see the new_module call which uses python's type() function to generate new classes dynamically.<br /> <br /> <br /> === How do modules deal with multiple inputs in a same port? ===<br /> <br /> (And should that even be allowed?)<br /> <br /> For compatibility reasons, we do need to allow multiple connections to an input port. However, most package developers should never have to use this, and so we do our best to hide it. the default behavior for getting inputs from a port, then, is to always return a single input.<br /> <br /> If on your module you need multiple inputs connected to a single port, use the 'forceGetInputListFromPort' method. It will return a list of all the data items coming through the port. The VTK package uses this feature, so look there for usage examples (packages/vtk/base_widget.py)<br /> <br /> === Are there mechanisms for attaching widgets to different modules/parameters? ===<br /> <br /> Right now, we have a mechanism for putting a specific widget for an input port. For example, if a port is SetColor(red, green, blue), we can put a color wheel widget there. Or we can also replace the SetFileName port with a File Widget. However, this is not per parameter (only per port). We are currently working on this problem.<br /> <br /> === Can I organize my package so it appears hierarchical in the module palette? ===<br /> <br /> Yes. Use the namespace keyword argument when adding the module to the registry. For example,<br /> <br /> registry.add_module(MyModule, namespace='MyNamespace')<br /> <br /> === Can I nest namespaces? ===<br /> <br /> Yes. Use the '|' character to separate different the hierarchy. For example,<br /> <br /> registry.add_module(MyModule, namespace='ParentNamespace|ChildNamespace')<br /> <br /> === Are there shortucts for registry initialization? ===<br /> <br /> Yes. If you define _modules as a list of classes in the __init__.py file of your package, VisTrails will attempt to load all classes specified as modules. You can provide add_module options as keyword arguments by specifying a tuple (class, kwargs) in the list. For example:<br /> <br /> _modules = [MyModule1, (MyModule2, {'namespace': 'MyNamespace'})]<br /> <br /> In addition, you need to identify the ports of your modules as a field in your class by defining _input_ports and _output_ports lists. Here, the items in each list must be tuples of the form (portName, portSignature, optional=False, sort_key=-1). For example:<br /> <br /> class MyModule(Module):<br /> def compute(self):<br /> pass<br /> <br /> _input_ports = [('firstInput', String), ('secondInput', Integer, True)]<br /> _output_ports = [('firstOutput', String), ('secondOutput', String)]<br /> <br /> === Can I define ports to be of types that I do not import into my package? ===<br /> <br /> Yes. You can pass an identifier string as the portSignature instead. The port_signature string is defined by:<br /> <br /> <module_string> := <package_identifier>:[<namespace>|]<module_name>,<br /> <port_signature> := (<module_string>*)<br /> <br /> For example,<br /> <br /> registry.add_input_port(MyModule, 'myInputPort', '(edu.utah.sci.vistrails.basic:String)')<br /> <br /> or<br /> <br /> _input_ports = [('myInputPort', '(edu.utah.sci.vistrails.basic:String)')]<br /> <br /> === What do I need to change in my package to make it reloadable (new in v1.4.2)? ===<br /> See [http://www.vistrails.org/index.php/UsersGuideVisTrailsPackages#How_to_make_you_package_reloadable UsersGuideVisTrailsPackages] for an explanation.<br /> <br /> === Can I add default values or labels for parameters? ===<br /> <br /> Yes. Versions 1.4 and greater support these features. See [[UsersGuideVisTrailsPackages#Adding_default_values_and.2For_labels_for_parameters | UsersGuideVisTrailsPacakges]] for more details.<br /> <br /> <br /> === I want to write a module to load HDF data whose output (e.g., data, string) varies according to the input I give it. Is is possible to do this in VisTrails, and if yes, how can I do that? Ideally, I would like to avoid having to change the connection of my output every time I change the input. ===<br /> <br /> There are a few ways to tackle this - each has it's own benefits and pitfalls. Firstly, module connections do respect class hierarchies as we're familiar with in object oriented languages. For instance, A module can output a Constant of which String, Float, Integer, etc are specifications. In this way, you can have a subclass of something like HDFData be passed out of the module and the connections will be established regardless of the sub-type. This is a bit dangerous though. Modules downstream of such a class may not really know how to operate on certain types derived from the super-class. Extreme care must be taken both when creating the modules as well as connecting them to prevent things like this from happening.<br /> <br /> A second method that I employ in several different packages is the idea of a container class. For instance, the NumSciPy package uses a relatively generic container "Numpy Array" to encapsulate the data. Of course, these encapsulating objects can store dictionaries that other modules can easily access and understand how to operate on. Although this method is slightly more work, the benefits of a stricter typing of ports is beneficial - particularly upon interfacing with other packages that may depend on strongly typed constants (for example).<br /> <br /> == The Console ==<br /> <br /> === Where should I go to find out what I can call from the console and how to import it? ===<br /> <br /> We have tried to make some methods more accessible in the console via an api. You can import the api via <code>import api</code> in the console and see the available methods with <code>dir(api)</code>. To open a vistrail:<br /> <br /> import api<br /> api.open_vistrail_from_file('/Applications/VisTrails/examples/terminator.vt')<br /> <br /> To execute a version of a workflow, you currently have to go through the controller:<br /> <br /> api.select_version('Histogram')<br /> api.get_current_controller().execute_current_workflow()<br /> <br /> Currently, only a subset of VisTrails functionality is directly available from the api. However, since VisTrails is written in python, you can dig down starting with the VistrailsApplication or controller object to expose most of our internal methods. If you have suggestions for calls to be added to the api, please let us know.<br /> <br /> One other feature that we're working on, but is still in progress is the ability to construct workflows via the console. For example:<br /> <br /> vtk = load_package('edu.utah.sci.vistrails.vtk')<br /> vtk.vtkDataSetReader() # adds a vtkDataSetReader module to the pipeline<br /> # click on the new module<br /> a = selected_modules()[0] # get the one currently selected module<br /> a.SetFile('/vistrails/examples/data/head120.vtk') # sets the SetFile parmaeter for the data set reader<br /> b = vtk.vtkContourFilter() # adds a vtkContourFilter module to the pipeline and saves to var b<br /> b.SetInputConnection0(a.GetOutputPort0()) # connects a's GetOutputPort0 port to b's SetInputConnection0<br /> <br /> == VTK ==<br /> <br /> === Given a VTK visualization, how can I generate a webpage from it? ===<br /> <br /> Check out the html pipeline in offscreen.xml.<br /> <br /> === I'm trying to use VTK, but there doesn't seem to be any output. What is wrong? ===<br /> <br /> To use VTK on VisTrails, you need a slightly different way of connecting the renderer modules. Instead of using the standard RenderWindow/RenderWindowInteractor infrastructure, you simply connect the renderer to a VTKCell. The examples directory in the distribution has several VTK examples that illustrate.<br /> <br /> === I am trying to add a module to the workflow via Python, but how can I access vtk modules? ===<br /> <br /> Here's an example:<br /> <br /> import api<br /> <br /> vtvtk = 'edu.utah.sci.vistrails.vtk'<br /> <br /> module = api.add_module(0, 0, vtvtk, 'vtkContourFilter', '')<br /> <br /> <br /> The third argument in add_module is the package identifier. You can find this in the "Module Packages" panel of the Preferences; just click on the package you're interested in and it will appear in the information on the right.<br /> <br /> == matplotlib ==<br /> <br /> === I'm experiencing a problem with Latex labels and the matplotlib that comes with VisTrails 1.5. The script below entered to the interpreter that comes with VT is sufficient to reproduce it. ===<br /> <br /> import matplotlib.pyplot as plt<br /> plt.plot([1,2,3],[1,2,3])<br /> plt.xlabel("$foo$")<br /> <br /> Remove your ~/.matplotlib folder and re-start VisTrails<br /> <br /> == VisTrails Development ==<br /> <br /> === I would like to build VisTrails from source. Are there instructions on how to do this? ===<br /> <br /> Yes! Take a look at http://www.vistrails.org/index.php/Mac_Intel_Instructions</div>

Mates https://www.vistrails.org//index.php?title=FAQ&diff=3734 FAQ 2010-12-06T13:42:21Z

<p>Mates: /* = How do I execute workflows and control VisTrails through another application? */</p> <hr /> <div>Also check our [[known_issues|Known Issues]] page for troubleshooting.<br /> <br /> ----<br /> == Running workflows ==<br /> <br /> === How can I run a workflow using the command line? ===<br /> <br /> ('''Updated for version 1.2''') Call vistrails using the following options:<br /> <br /> <code>python vistrails.py -b path_to_vistrails_file:pipeline </code><br /> <br /> where pipeline can be a version '''tag name''' or version '''id'''<br /> <br /> '''NOTE:''' If you downloaded the MacOS X bundle, you can run vistrails from the command line via the following commands in the Terminal. Change the current directory to wherever VisTrails was installed (often <code>/Applications</code>), and then type:<br /> <br /> <code>Vistrails.app/Contents/MacOS/vistrails [<cmd_line_options>]</code><br /> <br /> === Using the command line, we'd like to execute a workflow multiple times, with slightly different parameters, and create a series of output files. Is this possible?===<br /> <br /> ('''Updated for version 1.2''') We can change parameters that have an alias through the command line.<br /> <br /> For example, offscreen pipeline in offscreen.vt always creates the file called image.png. If you want generate it with a different filename:<br /> <br /> <code>python vistrails.py -b ../examples/offscreen.vt:offscreen -a"filename=other.png"</code><br /> <br /> <code>filename</code> in the example above is the alias name assigned to the parameter in the <code>value</code> method inside the String module. When running a pipeline from the command line, VisTrails will try to start the spreadsheet automatically if the pipeline requires it. For example, this other execution will also start the spreadsheet:<br /> <br /> <code>python vistrails.py -b ../examples/head.vt:aliases \<br /> -a"isovalue=30&&Diffuse_Color_R=0.8&&Diffuse_Color_G=0.4&&Diffuse_Color_B=0.2"</code><br /> <br /> You can also execute more than one pipeline on the command line:<br /> <br /> <code>python vistrails.py -b ../examples/head.vt:aliases ../examples/spx.vt:spx \<br /> -a"isovalue=30"</code><br /> <br /> Use the -a parameter only once regardless the number of pipelines.<br /> <br /> === I can load a vistrail, and the version tree shows up fine. However, no pipelines appear when I click on a version. What gives? ===<br /> <br /> The most likely reason is that the vistrail uses a package that is not registered with VisTrails. You need to identify the needed package and add it to your <code>.vistrails/startup.py</code>. A single line like the following should be enough:<br /> <br /> addPackage('enter_package_name_here')<br /> <br /> Some packages might need more information. For example:<br /> <br /> addPackage('afront', executable_path='/path/to/afront')<br /> <br /> Refer to the package documentation for details. The one inconvenient step is that currently there's no automated way to describe what is the missing package. We're working on this feature for future releases.<br /> <br /> === I have a workflow that reads a file and then does some processing. The first time it runs, it executes correctly. But in subsequent, nothing happens. ===<br /> <br /> VisTrails caches by default, so after a workflow is executed, if none of its parameters change, it won't be executed again. <br /> <br /> If a workflow reads a file using the basic module File, VisTrails does check whether the file was modified since the last run. It does so by keeping a signature that is based on the modification time of the file. And if the file was modified, the File module and all downstream modules (the ones which depend on File) will be executed. <br /> <br /> <br /> ''Note: If you would like your input and output data to be versioned, you can use the Persistence package.''<br /> <br /> If you do not want VisTrails to cache executions, you can turn off caching: go to Menu Edit -> Preferences and in the General Configuration tab, change Cache execution results to Never.<br /> <br /> === Can VisTrails execute workflows in parallel? ===<br /> <br /> The VisTrails server can only execute pipelines in parallel if there's more than one instance of VisTrails running. The command <br /> <br /> <code> self.rpcserver = ThreadedXMLRPCServer((self.temp_xml_rpc_options.server, self.temp_xml_rpc_options.port))</code><br /> <br /> starts a multithreaded version of the XML-RPC server, so it will create a thread for each request received by the server. The problem is that Qt/PyQT doesn't allow these multiple threads create GUI objects, only in the main thread. To overcome this limitation, the multithreaded version can instantiate other single threaded versions of VisTrails and put them in a queue, so workflow executions and other GUI-related requests, such as generating workflow graphs and history trees can be forwarded to this queue, and each instance takes turns in answering the request. If the results are in the cache, the multithreaded version answers the requests directly.<br /> <br /> Note that this infrastructure works on Linux only. To make this work on Windows, you have to create a script similar to start_vistrails_xvfb.sh (located in the scripts folder) where you can send the number of other instances via command-line options to VisTrails. The command line options are:<br /> <br /> python vistrails_server.py -T <ADDRESS> -R <PORT> -O<NUMBER_OF_OTHER_VISTRAILS_INSTANCES> [-M]&<br /> <br /> If you want the main vistrails instance to be multithreaded, use the -M at the end.<br /> <br /> After creating this script, update function start_other_instances in vistrails/gui/application_server.py lines 1007-1023 and set the script variable to point to your script. You may also have to change the arguments sent to your script (line 1016: for example, you don't need to set a virtual display). You will need to change the path to the stop_vistrails_server.py script (on line 1026) according to your installation path.<br /> <br /> === When a workflow is executed, what do the colors mean? ===<br /> <br /> - lilac: module was notexecuted<br /> <br /> - yellow: module is currently being executed <br /> <br /> - green: module was successfully executed <br /> <br /> - orange: module was cached<br /> <br /> - red: the execution of the module failed <br /> <br /> == Building workflows ==<br /> <br /> === Is there a way to give each widget a "display name" in addition to the module name at the center of the widget? ===<br /> <br /> Yes, but it is not easily accessible from the GUI and it definitely needs to be more intuitive. For now, we use the annotation value of key "__desc__" as a module label. If you want to set a PythonSource label, you have to select the module. Then click on the Annotation tab, and add a key named "__desc__", whatever value you set to this key will be the label. We are currently working on a new interface for this functionality.<br /> <br /> === Is there a way to re-center the picture-in-picture (PiP) view? ===<br /> <br /> Yes. If you click on the PIP window to bring it to focus, you can press Ctrl-R (or Command-R on Mac) to re-center the PiP window.<br /> <br /> === How do I search for a literal "?" (question mark) in the search box in the Property panel? ===<br /> <br /> Since we allow regular expressions in our search box, question marks are treated as meta-characters. Thus, searching for "?" returns everything and "abc?" will return everything containing "abc". You need to use "\?" instead to search for "?". So the search for "??" would be "\?\?".<br /> <br /> == Using VisTrails as a server ==<br /> <br /> === What is the VisTrails server-mode? ===<br /> <br /> Using the VisTrails server mode, it is possible to ''execute workflows and control VisTrails through another application''. For example, the CrowdLabs Web portal (http://www.crowdlabs.org) accesses a VisTrails sever to execute workflows, retrieve and display vistrail trees and workflows.<br /> <br /> === How do I execute workflows and control VisTrails through another application? ==<br /> <br /> The way you access the server is by doing XML-RPC calls. In the current VisTrails release, we include a set of PHP scripts that can talk to a VisTrails server instance. They are in "extensions/http" folder. The files are reasonably well documented. Also, it should be not difficult to create python scripts to access the server (just use xmlrpclib module).<br /> <br /> Note that the VisTrails server requires the provenance and workflows to be in a database. More detailed instructions on how to setup the server and the database<br /> are available here:<br /> <br /> http://www.crowdlabs.org/site_media/static/dev_docs/vistrails_server_setup.html<br /> <br /> http://www.crowdlabs.org/site_media/static/dev_docs/vistrails_database_setup.html<br /> <br /> If what you want is just to execute a series of workflows in batch mode, a simpler solution would be to ''use the VisTrails client in batch mode''. Chapter 12 of the user's guide contains detailed information and examples on that.<br /> <br /> == Control Flow ==<br /> <br /> === Note: using map ===<br /> <br /> When using 'map', the module (or subworkflow) used as function port in the map module MUST be a function, i.e., it can only define 1 output port. <br /> <br /> == Spreadsheet ==<br /> <br /> Where ''pipeline'' is a version number or a tag.<br /> <br /> === How can I save an image from the spreadsheet? ===<br /> <br /> While having the focus on a spreadsheet cell and select the camera on the toolbar to take a snapshot. The system will prompt you for the location and file name where it should be saved. The other icons can be used for saving multiple images that can be used for generating an animation on demand. A whole sheet can also be saved by selecting Export (either from the menu or from the toolbar).<br /> <br /> ===Is it possible to save the complete state of the spreadsheet?===<br /> <br /> === Can I view multiple sheets at the same time? ===<br /> <br /> Yes. Each sheet on the spreadsheet can be displayed as a dock widget separated from the main spreadsheet window by dragging its tab name out of the tab bar at the bottom of the spreadsheet.<br /> <br /> === Then, how can I put back a separated sheet? ===<br /> <br /> A sheet can be docked back to the main window by dragging it back to the tab bar or double-click on its title bar.<br /> <br /> === How can I order sheets on the spreadsheet? ===<br /> <br /> This can be done by dragging the sheet name on the bottom top bar and drop it to the right place.<br /> <br /> === Can I control where a cell will be placed on the spreadsheet window? ===<br /> <br /> By default, an unoccupied cell on the active sheet will be chosen to display the result. However, you can specify exactly in the pipeline where a spreadsheet cell will be placed by using CellLocation and SheetReference. CellLocation specifies the location (row and column) of a cell when connecting to a spreadsheet cell (VTKCell, ImageViewerCell, ...). Similarly, a SheetReference module (when connecting to a CellLocation) will specify which sheet the cell will be put on given its name, minimum row size and minimum column size. There is an example of this in examples/vtk.xml (select the version below Double Renderer).<br /> <br /> === How do I output results to the spreadsheet? ===<br /> <br /> By inspecting the VisTrails Spreadsheet package (in the list of packages, to the left of the pipeline builder), you<br /> can see there are built-in cells for different kinds of data, e.g., RichTextCell to display HTML and plain text. op<br /> You (the user) can also define new cell types to display application-specific data. For example, we have developed<br /> VtkCell, MplFigureCell, and OpenGLCell. It is possible to display pretty much anything on the Spreadsheet!<br /> <br /> Examples of writing cell modules can be found in:<br /> RichTextCell: packages/spreadsheet/widgets/richtext/richtext.py<br /> VTK: packages/vtk/vtkcell.py<br /> <br /> Here is the summary of some requirements on a cell widget:<br /> <br /> (1) It must be a Qt widget. It should inherit from spreadsheet_cell.QCellWidget in the spreadsheet package. Although any Qt Widget would work, certain features such as animation will not be available (without rewriting it).<br /> <br /> (2) It must re-implement the updateContents() function to take a set of inputs (usually coming from input ports of a wrapper Module) and display on the cells. VisTrails uses this function to update/reuse cells on the spreadsheet when new data comes in.<br /> <br /> (3) It needs a wrapper VisTrails Module (inherited from basic_widgets.SpreadsheetCell of the spreadsheet package). Inside the compute() method of this module, it may call self.display(CellWidgetType, (inputs)) to trigger the display event on the spreadsheet.<br /> <br /> === How do I control the default number of cells in the spreadsheet? ===<br /> <br /> You can configure the rowCount and colCount using the preferences dialog. Just go to the Module Packages tab, select spreadsheet in the "Enabled packages" and press the Configure button. Then a list of all the configuration options for the spreadsheet will show up.<br /> <br /> === Is it possible to launch a web browser from the vistrails spreadsheet? We would like to output several urls from a parameter sweep and then have the option to click on each one to view the resulting page. I can view the page within the spreadsheet, but it is really too crowded.===<br /> <br /> Currently, there isn't a widget that provides exactly this functionality, but I can think of a few solutions that may work for you:<br /> <br /> (1) You can use parameter exploration to generate multiple sheets so you might have an exploration that opens each page in a new sheet. Use the third column/dimension in the exploration interface to have a parameter span sheets.<br /> <br /> (2) The spreadsheet is extensible so you can write a custom spreadsheet cell widget that has a button or label with the desired link (a QLabel with openExternalLinks set to True, for example).<br /> <br /> (3) You can tweak the existing RichTextCell be adding the line "self.browser.setOpenExternalLinks(True)" at line 63 of the source file "vistrails/packages/spreadsheet/widgets/richtext/richtext.py". Then, if your workflow creates a file with html markup text like "<a href="http://www.vistrails.org">VisTrails</a>" connected to a RichTextCell, clicking on the rendered link in the cell will open it in a web browser. You need to add the aforementioned line to the source to let Qt know that you want the link opened externally; by default, it will just issue an event that isn't processed.<br /> <br /> == Integrating your software into VisTrails ==<br /> <br /> ===How can I integrate my own program into VisTrails?===<br /> <br /> The easiest way is to create a package. Writing a package is often very simple, here are instructions on how to do it: [[UsersGuideVisTrailsPackages]]<br /> <br /> You can also dynamically generate modules. For an example see:<br /> <br /> http://www.vistrails.org/index.php/UsersGuideVisTrailsPackages#Packages_that_generate_modules_dynamically<br /> <br /> In particular, see the new_module call which uses python's type() function to generate new classes dynamically.<br /> <br /> <br /> === How do modules deal with multiple inputs in a same port? ===<br /> <br /> (And should that even be allowed?)<br /> <br /> For compatibility reasons, we do need to allow multiple connections to an input port. However, most package developers should never have to use this, and so we do our best to hide it. the default behavior for getting inputs from a port, then, is to always return a single input.<br /> <br /> If on your module you need multiple inputs connected to a single port, use the 'forceGetInputListFromPort' method. It will return a list of all the data items coming through the port. The VTK package uses this feature, so look there for usage examples (packages/vtk/base_widget.py)<br /> <br /> === Are there mechanisms for attaching widgets to different modules/parameters? ===<br /> <br /> Right now, we have a mechanism for putting a specific widget for an input port. For example, if a port is SetColor(red, green, blue), we can put a color wheel widget there. Or we can also replace the SetFileName port with a File Widget. However, this is not per parameter (only per port). We are currently working on this problem.<br /> <br /> === Can I organize my package so it appears hierarchical in the module palette? ===<br /> <br /> Yes. Use the namespace keyword argument when adding the module to the registry. For example,<br /> <br /> registry.add_module(MyModule, namespace='MyNamespace')<br /> <br /> === Can I nest namespaces? ===<br /> <br /> Yes. Use the '|' character to separate different the hierarchy. For example,<br /> <br /> registry.add_module(MyModule, namespace='ParentNamespace|ChildNamespace')<br /> <br /> === Are there shortucts for registry initialization? ===<br /> <br /> Yes. If you define _modules as a list of classes in the __init__.py file of your package, VisTrails will attempt to load all classes specified as modules. You can provide add_module options as keyword arguments by specifying a tuple (class, kwargs) in the list. For example:<br /> <br /> _modules = [MyModule1, (MyModule2, {'namespace': 'MyNamespace'})]<br /> <br /> In addition, you need to identify the ports of your modules as a field in your class by defining _input_ports and _output_ports lists. Here, the items in each list must be tuples of the form (portName, portSignature, optional=False, sort_key=-1). For example:<br /> <br /> class MyModule(Module):<br /> def compute(self):<br /> pass<br /> <br /> _input_ports = [('firstInput', String), ('secondInput', Integer, True)]<br /> _output_ports = [('firstOutput', String), ('secondOutput', String)]<br /> <br /> === Can I define ports to be of types that I do not import into my package? ===<br /> <br /> Yes. You can pass an identifier string as the portSignature instead. The port_signature string is defined by:<br /> <br /> <module_string> := <package_identifier>:[<namespace>|]<module_name>,<br /> <port_signature> := (<module_string>*)<br /> <br /> For example,<br /> <br /> registry.add_input_port(MyModule, 'myInputPort', '(edu.utah.sci.vistrails.basic:String)')<br /> <br /> or<br /> <br /> _input_ports = [('myInputPort', '(edu.utah.sci.vistrails.basic:String)')]<br /> <br /> === What do I need to change in my package to make it reloadable (new in v1.4.2)? ===<br /> See [http://www.vistrails.org/index.php/UsersGuideVisTrailsPackages#How_to_make_you_package_reloadable UsersGuideVisTrailsPackages] for an explanation.<br /> <br /> === Can I add default values or labels for parameters? ===<br /> <br /> Yes. Versions 1.4 and greater support these features. See [[UsersGuideVisTrailsPackages#Adding_default_values_and.2For_labels_for_parameters | UsersGuideVisTrailsPacakges]] for more details.<br /> <br /> <br /> === I want to write a module to load HDF data whose output (e.g., data, string) varies according to the input I give it. Is is possible to do this in VisTrails, and if yes, how can I do that? Ideally, I would like to avoid having to change the connection of my output every time I change the input. ===<br /> <br /> There are a few ways to tackle this - each has it's own benefits and pitfalls. Firstly, module connections do respect class hierarchies as we're familiar with in object oriented languages. For instance, A module can output a Constant of which String, Float, Integer, etc are specifications. In this way, you can have a subclass of something like HDFData be passed out of the module and the connections will be established regardless of the sub-type. This is a bit dangerous though. Modules downstream of such a class may not really know how to operate on certain types derived from the super-class. Extreme care must be taken both when creating the modules as well as connecting them to prevent things like this from happening.<br /> <br /> A second method that I employ in several different packages is the idea of a container class. For instance, the NumSciPy package uses a relatively generic container "Numpy Array" to encapsulate the data. Of course, these encapsulating objects can store dictionaries that other modules can easily access and understand how to operate on. Although this method is slightly more work, the benefits of a stricter typing of ports is beneficial - particularly upon interfacing with other packages that may depend on strongly typed constants (for example).<br /> <br /> == The Console ==<br /> <br /> === Where should I go to find out what I can call from the console and how to import it? ===<br /> <br /> We have tried to make some methods more accessible in the console via an api. You can import the api via <code>import api</code> in the console and see the available methods with <code>dir(api)</code>. To open a vistrail:<br /> <br /> import api<br /> api.open_vistrail_from_file('/Applications/VisTrails/examples/terminator.vt')<br /> <br /> To execute a version of a workflow, you currently have to go through the controller:<br /> <br /> api.select_version('Histogram')<br /> api.get_current_controller().execute_current_workflow()<br /> <br /> Currently, only a subset of VisTrails functionality is directly available from the api. However, since VisTrails is written in python, you can dig down starting with the VistrailsApplication or controller object to expose most of our internal methods. If you have suggestions for calls to be added to the api, please let us know.<br /> <br /> One other feature that we're working on, but is still in progress is the ability to construct workflows via the console. For example:<br /> <br /> vtk = load_package('edu.utah.sci.vistrails.vtk')<br /> vtk.vtkDataSetReader() # adds a vtkDataSetReader module to the pipeline<br /> # click on the new module<br /> a = selected_modules()[0] # get the one currently selected module<br /> a.SetFile('/vistrails/examples/data/head120.vtk') # sets the SetFile parmaeter for the data set reader<br /> b = vtk.vtkContourFilter() # adds a vtkContourFilter module to the pipeline and saves to var b<br /> b.SetInputConnection0(a.GetOutputPort0()) # connects a's GetOutputPort0 port to b's SetInputConnection0<br /> <br /> == VTK ==<br /> <br /> === Given a VTK visualization, how can I generate a webpage from it? ===<br /> <br /> Check out the html pipeline in offscreen.xml.<br /> <br /> === I'm trying to use VTK, but there doesn't seem to be any output. What is wrong? ===<br /> <br /> To use VTK on VisTrails, you need a slightly different way of connecting the renderer modules. Instead of using the standard RenderWindow/RenderWindowInteractor infrastructure, you simply connect the renderer to a VTKCell. The examples directory in the distribution has several VTK examples that illustrate.<br /> <br /> === I am trying to add a module to the workflow via Python, but how can I access vtk modules? ===<br /> <br /> Here's an example:<br /> <br /> import api<br /> <br /> vtvtk = 'edu.utah.sci.vistrails.vtk'<br /> <br /> module = api.add_module(0, 0, vtvtk, 'vtkContourFilter', '')<br /> <br /> <br /> The third argument in add_module is the package identifier. You can find this in the "Module Packages" panel of the Preferences; just click on the package you're interested in and it will appear in the information on the right.<br /> <br /> == matplotlib ==<br /> <br /> === I'm experiencing a problem with Latex labels and the matplotlib that comes with VisTrails 1.5. The script below entered to the interpreter that comes with VT is sufficient to reproduce it. ===<br /> <br /> import matplotlib.pyplot as plt<br /> plt.plot([1,2,3],[1,2,3])<br /> plt.xlabel("$foo$")<br /> <br /> Remove your ~/.matplotlib folder and re-start VisTrails<br /> <br /> == VisTrails Development ==<br /> <br /> === I would like to build VisTrails from source. Are there instructions on how to do this? ===<br /> <br /> Yes! Take a look at http://www.vistrails.org/index.php/Mac_Intel_Instructions</div>

Mates https://www.vistrails.org//index.php?title=FAQ&diff=3733 FAQ 2010-12-06T13:42:06Z

<p>Mates: /* What is the VisTrails server-mode? */</p> <hr /> <div>Also check our [[known_issues|Known Issues]] page for troubleshooting.<br /> <br /> ----<br /> == Running workflows ==<br /> <br /> === How can I run a workflow using the command line? ===<br /> <br /> ('''Updated for version 1.2''') Call vistrails using the following options:<br /> <br /> <code>python vistrails.py -b path_to_vistrails_file:pipeline </code><br /> <br /> where pipeline can be a version '''tag name''' or version '''id'''<br /> <br /> '''NOTE:''' If you downloaded the MacOS X bundle, you can run vistrails from the command line via the following commands in the Terminal. Change the current directory to wherever VisTrails was installed (often <code>/Applications</code>), and then type:<br /> <br /> <code>Vistrails.app/Contents/MacOS/vistrails [<cmd_line_options>]</code><br /> <br /> === Using the command line, we'd like to execute a workflow multiple times, with slightly different parameters, and create a series of output files. Is this possible?===<br /> <br /> ('''Updated for version 1.2''') We can change parameters that have an alias through the command line.<br /> <br /> For example, offscreen pipeline in offscreen.vt always creates the file called image.png. If you want generate it with a different filename:<br /> <br /> <code>python vistrails.py -b ../examples/offscreen.vt:offscreen -a"filename=other.png"</code><br /> <br /> <code>filename</code> in the example above is the alias name assigned to the parameter in the <code>value</code> method inside the String module. When running a pipeline from the command line, VisTrails will try to start the spreadsheet automatically if the pipeline requires it. For example, this other execution will also start the spreadsheet:<br /> <br /> <code>python vistrails.py -b ../examples/head.vt:aliases \<br /> -a"isovalue=30&&Diffuse_Color_R=0.8&&Diffuse_Color_G=0.4&&Diffuse_Color_B=0.2"</code><br /> <br /> You can also execute more than one pipeline on the command line:<br /> <br /> <code>python vistrails.py -b ../examples/head.vt:aliases ../examples/spx.vt:spx \<br /> -a"isovalue=30"</code><br /> <br /> Use the -a parameter only once regardless the number of pipelines.<br /> <br /> === I can load a vistrail, and the version tree shows up fine. However, no pipelines appear when I click on a version. What gives? ===<br /> <br /> The most likely reason is that the vistrail uses a package that is not registered with VisTrails. You need to identify the needed package and add it to your <code>.vistrails/startup.py</code>. A single line like the following should be enough:<br /> <br /> addPackage('enter_package_name_here')<br /> <br /> Some packages might need more information. For example:<br /> <br /> addPackage('afront', executable_path='/path/to/afront')<br /> <br /> Refer to the package documentation for details. The one inconvenient step is that currently there's no automated way to describe what is the missing package. We're working on this feature for future releases.<br /> <br /> === I have a workflow that reads a file and then does some processing. The first time it runs, it executes correctly. But in subsequent, nothing happens. ===<br /> <br /> VisTrails caches by default, so after a workflow is executed, if none of its parameters change, it won't be executed again. <br /> <br /> If a workflow reads a file using the basic module File, VisTrails does check whether the file was modified since the last run. It does so by keeping a signature that is based on the modification time of the file. And if the file was modified, the File module and all downstream modules (the ones which depend on File) will be executed. <br /> <br /> <br /> ''Note: If you would like your input and output data to be versioned, you can use the Persistence package.''<br /> <br /> If you do not want VisTrails to cache executions, you can turn off caching: go to Menu Edit -> Preferences and in the General Configuration tab, change Cache execution results to Never.<br /> <br /> === Can VisTrails execute workflows in parallel? ===<br /> <br /> The VisTrails server can only execute pipelines in parallel if there's more than one instance of VisTrails running. The command <br /> <br /> <code> self.rpcserver = ThreadedXMLRPCServer((self.temp_xml_rpc_options.server, self.temp_xml_rpc_options.port))</code><br /> <br /> starts a multithreaded version of the XML-RPC server, so it will create a thread for each request received by the server. The problem is that Qt/PyQT doesn't allow these multiple threads create GUI objects, only in the main thread. To overcome this limitation, the multithreaded version can instantiate other single threaded versions of VisTrails and put them in a queue, so workflow executions and other GUI-related requests, such as generating workflow graphs and history trees can be forwarded to this queue, and each instance takes turns in answering the request. If the results are in the cache, the multithreaded version answers the requests directly.<br /> <br /> Note that this infrastructure works on Linux only. To make this work on Windows, you have to create a script similar to start_vistrails_xvfb.sh (located in the scripts folder) where you can send the number of other instances via command-line options to VisTrails. The command line options are:<br /> <br /> python vistrails_server.py -T <ADDRESS> -R <PORT> -O<NUMBER_OF_OTHER_VISTRAILS_INSTANCES> [-M]&<br /> <br /> If you want the main vistrails instance to be multithreaded, use the -M at the end.<br /> <br /> After creating this script, update function start_other_instances in vistrails/gui/application_server.py lines 1007-1023 and set the script variable to point to your script. You may also have to change the arguments sent to your script (line 1016: for example, you don't need to set a virtual display). You will need to change the path to the stop_vistrails_server.py script (on line 1026) according to your installation path.<br /> <br /> === When a workflow is executed, what do the colors mean? ===<br /> <br /> - lilac: module was notexecuted<br /> <br /> - yellow: module is currently being executed <br /> <br /> - green: module was successfully executed <br /> <br /> - orange: module was cached<br /> <br /> - red: the execution of the module failed <br /> <br /> == Building workflows ==<br /> <br /> === Is there a way to give each widget a "display name" in addition to the module name at the center of the widget? ===<br /> <br /> Yes, but it is not easily accessible from the GUI and it definitely needs to be more intuitive. For now, we use the annotation value of key "__desc__" as a module label. If you want to set a PythonSource label, you have to select the module. Then click on the Annotation tab, and add a key named "__desc__", whatever value you set to this key will be the label. We are currently working on a new interface for this functionality.<br /> <br /> === Is there a way to re-center the picture-in-picture (PiP) view? ===<br /> <br /> Yes. If you click on the PIP window to bring it to focus, you can press Ctrl-R (or Command-R on Mac) to re-center the PiP window.<br /> <br /> === How do I search for a literal "?" (question mark) in the search box in the Property panel? ===<br /> <br /> Since we allow regular expressions in our search box, question marks are treated as meta-characters. Thus, searching for "?" returns everything and "abc?" will return everything containing "abc". You need to use "\?" instead to search for "?". So the search for "??" would be "\?\?".<br /> <br /> == Using VisTrails as a server ==<br /> <br /> === What is the VisTrails server-mode? ===<br /> <br /> Using the VisTrails server mode, it is possible to ''execute workflows and control VisTrails through another application''. For example, the CrowdLabs Web portal (http://www.crowdlabs.org) accesses a VisTrails sever to execute workflows, retrieve and display vistrail trees and workflows.<br /> <br /> === How do I execute workflows and control VisTrails through another application? ==<br /> <br /> The way you access the server is by doing XML-RPC calls. In the current VisTrails release, we include a set of PHP scripts that can talk to a VisTrails server instance. They are in "extensions/http" folder. The files are reasonably well documented. Also, it should be not difficult to create python scripts to access the server (just use xmlrpclib module).<br /> <br /> Note that the VisTrails server requires the provenance and workflows to be in a database. More detailed instructions on how to setup the server and the database<br /> are available here:<br /> <br /> http://www.crowdlabs.org/site_media/static/dev_docs/vistrails_server_setup.html<br /> <br /> http://www.crowdlabs.org/site_media/static/dev_docs/vistrails_database_setup.html<br /> <br /> If what you want is just to execute a series of workflows in batch mode, a simpler solution would be to ''use the VisTrails client in batch mode''. Chapter 12 of the user's guide contains detailed information and examples on that.<br /> <br /> == Control Flow ==<br /> <br /> === Note: using map ===<br /> <br /> When using 'map', the module (or subworkflow) used as function port in the map module MUST be a function, i.e., it can only define 1 output port. <br /> <br /> == Spreadsheet ==<br /> <br /> Where ''pipeline'' is a version number or a tag.<br /> <br /> === How can I save an image from the spreadsheet? ===<br /> <br /> While having the focus on a spreadsheet cell and select the camera on the toolbar to take a snapshot. The system will prompt you for the location and file name where it should be saved. The other icons can be used for saving multiple images that can be used for generating an animation on demand. A whole sheet can also be saved by selecting Export (either from the menu or from the toolbar).<br /> <br /> ===Is it possible to save the complete state of the spreadsheet?===<br /> <br /> === Can I view multiple sheets at the same time? ===<br /> <br /> Yes. Each sheet on the spreadsheet can be displayed as a dock widget separated from the main spreadsheet window by dragging its tab name out of the tab bar at the bottom of the spreadsheet.<br /> <br /> === Then, how can I put back a separated sheet? ===<br /> <br /> A sheet can be docked back to the main window by dragging it back to the tab bar or double-click on its title bar.<br /> <br /> === How can I order sheets on the spreadsheet? ===<br /> <br /> This can be done by dragging the sheet name on the bottom top bar and drop it to the right place.<br /> <br /> === Can I control where a cell will be placed on the spreadsheet window? ===<br /> <br /> By default, an unoccupied cell on the active sheet will be chosen to display the result. However, you can specify exactly in the pipeline where a spreadsheet cell will be placed by using CellLocation and SheetReference. CellLocation specifies the location (row and column) of a cell when connecting to a spreadsheet cell (VTKCell, ImageViewerCell, ...). Similarly, a SheetReference module (when connecting to a CellLocation) will specify which sheet the cell will be put on given its name, minimum row size and minimum column size. There is an example of this in examples/vtk.xml (select the version below Double Renderer).<br /> <br /> === How do I output results to the spreadsheet? ===<br /> <br /> By inspecting the VisTrails Spreadsheet package (in the list of packages, to the left of the pipeline builder), you<br /> can see there are built-in cells for different kinds of data, e.g., RichTextCell to display HTML and plain text. op<br /> You (the user) can also define new cell types to display application-specific data. For example, we have developed<br /> VtkCell, MplFigureCell, and OpenGLCell. It is possible to display pretty much anything on the Spreadsheet!<br /> <br /> Examples of writing cell modules can be found in:<br /> RichTextCell: packages/spreadsheet/widgets/richtext/richtext.py<br /> VTK: packages/vtk/vtkcell.py<br /> <br /> Here is the summary of some requirements on a cell widget:<br /> <br /> (1) It must be a Qt widget. It should inherit from spreadsheet_cell.QCellWidget in the spreadsheet package. Although any Qt Widget would work, certain features such as animation will not be available (without rewriting it).<br /> <br /> (2) It must re-implement the updateContents() function to take a set of inputs (usually coming from input ports of a wrapper Module) and display on the cells. VisTrails uses this function to update/reuse cells on the spreadsheet when new data comes in.<br /> <br /> (3) It needs a wrapper VisTrails Module (inherited from basic_widgets.SpreadsheetCell of the spreadsheet package). Inside the compute() method of this module, it may call self.display(CellWidgetType, (inputs)) to trigger the display event on the spreadsheet.<br /> <br /> === How do I control the default number of cells in the spreadsheet? ===<br /> <br /> You can configure the rowCount and colCount using the preferences dialog. Just go to the Module Packages tab, select spreadsheet in the "Enabled packages" and press the Configure button. Then a list of all the configuration options for the spreadsheet will show up.<br /> <br /> === Is it possible to launch a web browser from the vistrails spreadsheet? We would like to output several urls from a parameter sweep and then have the option to click on each one to view the resulting page. I can view the page within the spreadsheet, but it is really too crowded.===<br /> <br /> Currently, there isn't a widget that provides exactly this functionality, but I can think of a few solutions that may work for you:<br /> <br /> (1) You can use parameter exploration to generate multiple sheets so you might have an exploration that opens each page in a new sheet. Use the third column/dimension in the exploration interface to have a parameter span sheets.<br /> <br /> (2) The spreadsheet is extensible so you can write a custom spreadsheet cell widget that has a button or label with the desired link (a QLabel with openExternalLinks set to True, for example).<br /> <br /> (3) You can tweak the existing RichTextCell be adding the line "self.browser.setOpenExternalLinks(True)" at line 63 of the source file "vistrails/packages/spreadsheet/widgets/richtext/richtext.py". Then, if your workflow creates a file with html markup text like "<a href="http://www.vistrails.org">VisTrails</a>" connected to a RichTextCell, clicking on the rendered link in the cell will open it in a web browser. You need to add the aforementioned line to the source to let Qt know that you want the link opened externally; by default, it will just issue an event that isn't processed.<br /> <br /> == Integrating your software into VisTrails ==<br /> <br /> ===How can I integrate my own program into VisTrails?===<br /> <br /> The easiest way is to create a package. Writing a package is often very simple, here are instructions on how to do it: [[UsersGuideVisTrailsPackages]]<br /> <br /> You can also dynamically generate modules. For an example see:<br /> <br /> http://www.vistrails.org/index.php/UsersGuideVisTrailsPackages#Packages_that_generate_modules_dynamically<br /> <br /> In particular, see the new_module call which uses python's type() function to generate new classes dynamically.<br /> <br /> <br /> === How do modules deal with multiple inputs in a same port? ===<br /> <br /> (And should that even be allowed?)<br /> <br /> For compatibility reasons, we do need to allow multiple connections to an input port. However, most package developers should never have to use this, and so we do our best to hide it. the default behavior for getting inputs from a port, then, is to always return a single input.<br /> <br /> If on your module you need multiple inputs connected to a single port, use the 'forceGetInputListFromPort' method. It will return a list of all the data items coming through the port. The VTK package uses this feature, so look there for usage examples (packages/vtk/base_widget.py)<br /> <br /> === Are there mechanisms for attaching widgets to different modules/parameters? ===<br /> <br /> Right now, we have a mechanism for putting a specific widget for an input port. For example, if a port is SetColor(red, green, blue), we can put a color wheel widget there. Or we can also replace the SetFileName port with a File Widget. However, this is not per parameter (only per port). We are currently working on this problem.<br /> <br /> === Can I organize my package so it appears hierarchical in the module palette? ===<br /> <br /> Yes. Use the namespace keyword argument when adding the module to the registry. For example,<br /> <br /> registry.add_module(MyModule, namespace='MyNamespace')<br /> <br /> === Can I nest namespaces? ===<br /> <br /> Yes. Use the '|' character to separate different the hierarchy. For example,<br /> <br /> registry.add_module(MyModule, namespace='ParentNamespace|ChildNamespace')<br /> <br /> === Are there shortucts for registry initialization? ===<br /> <br /> Yes. If you define _modules as a list of classes in the __init__.py file of your package, VisTrails will attempt to load all classes specified as modules. You can provide add_module options as keyword arguments by specifying a tuple (class, kwargs) in the list. For example:<br /> <br /> _modules = [MyModule1, (MyModule2, {'namespace': 'MyNamespace'})]<br /> <br /> In addition, you need to identify the ports of your modules as a field in your class by defining _input_ports and _output_ports lists. Here, the items in each list must be tuples of the form (portName, portSignature, optional=False, sort_key=-1). For example:<br /> <br /> class MyModule(Module):<br /> def compute(self):<br /> pass<br /> <br /> _input_ports = [('firstInput', String), ('secondInput', Integer, True)]<br /> _output_ports = [('firstOutput', String), ('secondOutput', String)]<br /> <br /> === Can I define ports to be of types that I do not import into my package? ===<br /> <br /> Yes. You can pass an identifier string as the portSignature instead. The port_signature string is defined by:<br /> <br /> <module_string> := <package_identifier>:[<namespace>|]<module_name>,<br /> <port_signature> := (<module_string>*)<br /> <br /> For example,<br /> <br /> registry.add_input_port(MyModule, 'myInputPort', '(edu.utah.sci.vistrails.basic:String)')<br /> <br /> or<br /> <br /> _input_ports = [('myInputPort', '(edu.utah.sci.vistrails.basic:String)')]<br /> <br /> === What do I need to change in my package to make it reloadable (new in v1.4.2)? ===<br /> See [http://www.vistrails.org/index.php/UsersGuideVisTrailsPackages#How_to_make_you_package_reloadable UsersGuideVisTrailsPackages] for an explanation.<br /> <br /> === Can I add default values or labels for parameters? ===<br /> <br /> Yes. Versions 1.4 and greater support these features. See [[UsersGuideVisTrailsPackages#Adding_default_values_and.2For_labels_for_parameters | UsersGuideVisTrailsPacakges]] for more details.<br /> <br /> <br /> === I want to write a module to load HDF data whose output (e.g., data, string) varies according to the input I give it. Is is possible to do this in VisTrails, and if yes, how can I do that? Ideally, I would like to avoid having to change the connection of my output every time I change the input. ===<br /> <br /> There are a few ways to tackle this - each has it's own benefits and pitfalls. Firstly, module connections do respect class hierarchies as we're familiar with in object oriented languages. For instance, A module can output a Constant of which String, Float, Integer, etc are specifications. In this way, you can have a subclass of something like HDFData be passed out of the module and the connections will be established regardless of the sub-type. This is a bit dangerous though. Modules downstream of such a class may not really know how to operate on certain types derived from the super-class. Extreme care must be taken both when creating the modules as well as connecting them to prevent things like this from happening.<br /> <br /> A second method that I employ in several different packages is the idea of a container class. For instance, the NumSciPy package uses a relatively generic container "Numpy Array" to encapsulate the data. Of course, these encapsulating objects can store dictionaries that other modules can easily access and understand how to operate on. Although this method is slightly more work, the benefits of a stricter typing of ports is beneficial - particularly upon interfacing with other packages that may depend on strongly typed constants (for example).<br /> <br /> == The Console ==<br /> <br /> === Where should I go to find out what I can call from the console and how to import it? ===<br /> <br /> We have tried to make some methods more accessible in the console via an api. You can import the api via <code>import api</code> in the console and see the available methods with <code>dir(api)</code>. To open a vistrail:<br /> <br /> import api<br /> api.open_vistrail_from_file('/Applications/VisTrails/examples/terminator.vt')<br /> <br /> To execute a version of a workflow, you currently have to go through the controller:<br /> <br /> api.select_version('Histogram')<br /> api.get_current_controller().execute_current_workflow()<br /> <br /> Currently, only a subset of VisTrails functionality is directly available from the api. However, since VisTrails is written in python, you can dig down starting with the VistrailsApplication or controller object to expose most of our internal methods. If you have suggestions for calls to be added to the api, please let us know.<br /> <br /> One other feature that we're working on, but is still in progress is the ability to construct workflows via the console. For example:<br /> <br /> vtk = load_package('edu.utah.sci.vistrails.vtk')<br /> vtk.vtkDataSetReader() # adds a vtkDataSetReader module to the pipeline<br /> # click on the new module<br /> a = selected_modules()[0] # get the one currently selected module<br /> a.SetFile('/vistrails/examples/data/head120.vtk') # sets the SetFile parmaeter for the data set reader<br /> b = vtk.vtkContourFilter() # adds a vtkContourFilter module to the pipeline and saves to var b<br /> b.SetInputConnection0(a.GetOutputPort0()) # connects a's GetOutputPort0 port to b's SetInputConnection0<br /> <br /> == VTK ==<br /> <br /> === Given a VTK visualization, how can I generate a webpage from it? ===<br /> <br /> Check out the html pipeline in offscreen.xml.<br /> <br /> === I'm trying to use VTK, but there doesn't seem to be any output. What is wrong? ===<br /> <br /> To use VTK on VisTrails, you need a slightly different way of connecting the renderer modules. Instead of using the standard RenderWindow/RenderWindowInteractor infrastructure, you simply connect the renderer to a VTKCell. The examples directory in the distribution has several VTK examples that illustrate.<br /> <br /> === I am trying to add a module to the workflow via Python, but how can I access vtk modules? ===<br /> <br /> Here's an example:<br /> <br /> import api<br /> <br /> vtvtk = 'edu.utah.sci.vistrails.vtk'<br /> <br /> module = api.add_module(0, 0, vtvtk, 'vtkContourFilter', '')<br /> <br /> <br /> The third argument in add_module is the package identifier. You can find this in the "Module Packages" panel of the Preferences; just click on the package you're interested in and it will appear in the information on the right.<br /> <br /> == matplotlib ==<br /> <br /> === I'm experiencing a problem with Latex labels and the matplotlib that comes with VisTrails 1.5. The script below entered to the interpreter that comes with VT is sufficient to reproduce it. ===<br /> <br /> import matplotlib.pyplot as plt<br /> plt.plot([1,2,3],[1,2,3])<br /> plt.xlabel("$foo$")<br /> <br /> Remove your ~/.matplotlib folder and re-start VisTrails<br /> <br /> == VisTrails Development ==<br /> <br /> === I would like to build VisTrails from source. Are there instructions on how to do this? ===<br /> <br /> Yes! Take a look at http://www.vistrails.org/index.php/Mac_Intel_Instructions</div>

Mates https://www.vistrails.org//index.php?title=Development&diff=3724 Development 2010-12-02T14:42:12Z

<p>Mates: /* Dec 7, 2010 */</p> <hr /> <div>== Roadmap ==<br /> <br /> <br /> == Weekly Meetings ==<br /> <br /> === Dec 7, 2010 ===<br /> <br /> * On merging master and crowdlabs git branches<br /> ** some updates on the crowdlabs branch need to be merged/patched into the master branch<br /> ** how should this be done?<br /> <br /> * PythonSource editor<br /> ** modal or multiple window editors?<br /> <br /> === Nov 30, 2010 ===<br /> <br /> * Subworkflow Upgrades (Daniel)<br /> ** Should be completed finished tonight <br /> ** Local subworkflow upgrades should be working now<br /> ** Need to test these after the checkin<br /> ** Using a dummy controller<br /> <br /> * Assistant for control package (Daniel)<br /> ** Look at examples<br /> ** Try to make an assistant to modify workflow using currently selected modules as looping group<br /> <br /> * [[Improve error handling/reporting]]<br /> ** Tons of message boxes pop up when a workflow has a bunch of errors, UI response is bad after this...<br /> ** Update existing error messages<br /> ** Show Details rendering on Mac<br /> ** What to do when multiple errors occur in a single event?<br /> ** New dialog that shows "next message" but allows user to dismiss messages or view all in the messages window<br /> ** Color coding for messages: use gui/theme.py to configure: maybe try black for warning, gray for log level messages<br /> <br /> * Data and workflow browser (Nivan will give a status update)<br /> <br /> * Saving Explorations (Feature Request from ALPS)<br /> ** We already save parameter explorations with the workflow in the version tree. As is, we save only the last exploration that was done for a given workflow. If you ship the vistrail, the saved parameter explorations will be there. Is this sufficient?<br /> ** [MT] Yes, that takes us a long way. Could one add a button in the explorations to save a certain set of explorations as a new version, like you do with the camera angle in the spreadsheet? That way one could store more than one set of exploration parameters without forcing a new version to be created with every change?<br /> ** Eventually we should provide a selector in parameter exploration<br /> ** Fastest to just allow a null action new version (a bit clunky) but wouldn't require too many changes<br /> <br /> * Clean up/prune persistent directory<br /> ** What are the concrete use cases for this?<br /> ** [MT] Here are two concrete use cases that appeared over the last week:<br /> *** a project is done, the important results copied to an archive and I want to get rid of all its intermediate files, etc., to save space - but not those files associated with other projects. Or, similarly, I'm done testing some workflows, want to remove all (partially buggy or irrelevant) files, and then start with production runs.<br /> *** there was a problem in a workflow(e.g. a bug in HTTPFile module with binary files or a bug in some user code). I want to force a certain persisted directory to be "forgotten", so that I can repair the wrong files. Bumping the version number of the ALPS package would be one way, but that is overkill since it would immediately invalidate all persisted files. Removing the whole persistence directory is just as radical.<br /> ** just testing simple tutorials with small files leaves me with 2 GB of persistent data after 10 days, in 142 persistent directories or files. I need some way to clean this up that is less radical then removing all persistent directories and files. This could be done, by e.g. removing all those which do not occur in tagged or leaf workflows in a certain set of vistrails. Or marking those which occur in certain vistrails with certain labels, and then being able to prune all those with or without some label/tag. Another option might be having the ability to choose from various repositories.<br /> ** [DK] follow up on this<br /> <br /> * Improvements to VisTrails server communication with clients to better handle errors/crashes (Phillip)<br /> ** work in progress<br /> <br /> * Data Publishing with crowdLabs. (Phillip) <br /> ** Do we want data publishing capabilities in VisTrails and crowdLabs? '''YES!!!'''<br /> ** If so...<br /> *** Currently limited data provenance for data. HTTPFile module lacks meta data, RepoSync module has poor usability.<br /> *** Persistent Data works for experimental data. Can we use it for publishing data?<br /> **** Publishing data or storing data on the server and pushing back and forth?<br /> **** What is the current scheme for identifying data on the server?<br /> **** Do we need to do better than the ids, versions, hashes that the peristence package provides?<br /> *** Do we want to utilize data publishing software like [http://thedata.org/home The Dataverse Network] or maybe [http://www.fedora-commons.org/ Fedora Commons]?<br /> ** [ES] Feature request: Can RepoSync support a directory of files?<br /> *** [PM] It can't, and I'm not sure how add it. I also don't really like the current RepoSync interface. It's not obvious how your data is being handled, hence me bring up data publishing, so hopefully we can come up with something more usable/robust.<br /> ** persistence used more for exploratory<br /> ** want to focus on more archive<br /> ** move from exploratory data to published<br /> ** Dataverse: essentially a site to curate data, archive data, host their own site, each university has their own server, linked together<br /> ** persistent store integration: maybe use git to do this, can add to local repositories to move to git<br /> ** use some of the dataverse: standard hash schemes, etc.<br /> ** privacy: can users push workflows without exposing data<br /> ** permissions for who can check things out: download permissions, upload permissions<br /> ** look into setting up persistent store on <br /> <br /> * DisplayWall (Wendel)<br /> ** new version of vistrails receives rotate/scale messages much more slowly<br /> ** using the vtk version that is already there<br /> ** comparing VisTrails 1.2 with nightly src<br /> ** test when the messages are received on the clients<br /> ** [ES] Changed something in the VTKCell, using the Qt interactor from vtk now, just uncomment this (vtkcell.py file)<br /> ** some commands are missed; check why this is happening<br /> ** preferences switch? 2x2 not 2x3<br /> ** check what modifications exist in the client code that's not exactly 1.2?<br /> ** new version of vtk package changed some things<br /> ** update vtk upgrade: resetcamera and addrgbpoint => _# versions<br /> ** Demos on displaywall: Wendel has to keep both VisTrails running and stop work when visitors stop by unannounced<br /> <br /> * Relative paths in workflows, how to deal with them? (Emanuele)<br /> ** [ES] Updated all examples to use HTTPFile for these<br /> ** some of these had to change completely (color widget), kept tree the same format<br /> <br /> === Nov 16, 2010 ===<br /> <br /> * Data and workflow browser (Nivan)<br /> ** Check into git branch (Dave)<br /> ** Add data support to the vistrails, workflows, and execs already exposed in the browser<br /> ** Add thumbnails to the browser<br /> ** Bittorrent support?<br /> <br /> * Upgrade issues -- a wrap? (Daniel)<br /> ** Fixing some issues with namespaces<br /> ** Upgrades for subworkflows that are included with packages, need to save these upgrades with the vt file as a "local abstraction" to ensure provenance. May be able to tie these back to the package versions once the package is upgraded.<br /> <br /> * Test Suite<br /> ** Matthias has his own test suite now<br /> ** We should make sure that our tests are up-to-date and run them before releases<br /> <br /> * Web services on users' guide (Tommy)<br /> ** Users' guide needs only a few changes. One of the changes should be a note making it explicit that we are using a new library and the old one was deprecated.<br /> ** Do we want to update the 39-page guide as well? For now, just update the users' guide. Don't distribute this guide. Just include it in a zip file that includes the old version.<br /> <br /> * [[Improve error handling/reporting]] (Tommy)<br /> ** print statements, has a new window<br /> ** add a menu item to view message window and also keep all messages from the session<br /> ** look into writing QMessageBox errors and warning to message window<br /> ** write all messages to a single window, write critical messages in red?<br /> ** have a utility method that encapsulates the gui error display and the core.debug display (if not running gui, only write to core.debug)<br /> ** don't display stack trace in a qmessagebox? -- try to emulate the invalid pipeline with "Show Details" button.<br /> ** make it easier for users to report errors by having a "copy" or "send to developers" button to report the details of an error.<br /> <br /> * DisplayWall (Wendel)<br /> ** What examples should be in the SVN? If we decide to keep some examples in the svn, some of them use data that should be along.<br /> *** Indicate progress of HTTPFile<br /> *** Had to upgrade examples to work with new examples of VisTrails<br /> *** VisTrails can send progress information with another thread like crowdlabs can do<br /> *** Each machine has a .vistrails and needs to download own version of data, make a single .vistrails for all machines?<br /> *** All slave machines can use the server as a cache?<br /> ** DisplayWall Client on the Apple Store? I dont like the idea of sharing the client source code. And putting the app without login/password will allow anyone to access our displaywall.<br /> *** Add a password for this<br /> *** Do user management?<br /> <br /> * VTK package:<br /> ** Have two ResetCamera modules now<br /> ** Need to write upgrade code<br /> ** Only with vtk 5.7 right now<br /> ** Encode library version in the package version?<br /> ** For reproducibility, we probably want to stick to specific versions<br /> ** We still need some ability to run new versions of vtk for testing, new features, etc., too...<br /> <br /> * Relative paths in workflows, how to deal with them? (Emanuele)<br /> ** Should we use dataDirectory?<br /> ** Change examples to not use hard-coded files?<br /> ** Problem with workflows, not data files<br /> ** On Windows, use ../examples<br /> ** On Mac, use ../../../examples<br /> ** dataDirectory not used, I think<br /> ** create "$VT_EXAMPLES"?<br /> ** problem is using new versions of VisTrails, data may change so referencing old location will fail<br /> ** don't allow users to change this variable<br /> ** convert to use HTTPFile<br /> <br /> * Data files and the GIT Repository (Emanuele)<br /> ** make a separate repository or use the submodule git functionality<br /> ** go with submodule<br /> <br /> * Next release: What still needs to be done?<br /> ** high priority from ALPS list<br /> ** Dan's fixes to the upgrade workflow<br /> ** Date for release? Official release in beginning of December for ALPS<br /> ** All developers should run test suite on push<br /> ** Could make this somewhat automatic, run with nightly release script and email -dev list<br /> ** Could have different classes of tests, make sure that critical ones pass<br /> ** Tests have a bunch of errors currently<br /> <br /> * Saving explorations:<br /> ** Keeps track of the latest exploration for each version<br /> ** Check with Matthias what the requirements for this are<br /> ** Have way to send just specification to a server (run this workflow with this parameter exploration)?<br /> ** Can we have an xml spec that specifies the parameter exploration (export parameter exploration)<br /> ** Currently persisting via xml and then converting to string so we already have an xml serialization<br /> ** Dan wrote API that takes xml string and populates the param_exp gui with the appropriate values<br /> ** Need to back the GUI state with core state to enable short-circuiting the gui<br /> <br /> <br /> * Box input ports should be colored (e.g. black) if they are already set by an internal method. Another color (e.g. red) could be used to indicate mandatory input ports.<br /> ** Currently, non-optional ports are square, shown. Optionals are circles and must be enabled<br /> ** Add a third category for mandatory, problem is backward compat.<br /> ** [ES] Colors are already overloaded, not for ports, use shading instead?<br /> <br /> === Nov 9, 2010 ===<br /> <br /> * FAQ<br /> ** As we reply to users queries, let's add the question and answer to the FAQ!<br /> ** Just a reminder<br /> <br /> * Caching of File module<br /> ** There was a message from a user who got confused because this module is cached by default (and silently). And in his application, since the file actually changed in between runs, he did not see the 'expected' result.<br /> ** Should we keep a hash for files and check whether they changed?<br /> ** Should we no cache File by default?<br /> ** [DK] File is supposed to have a special signature computation that detects changes in the file contents so this may be a bug instead. It does work, it seems to be an issue with the create_file<br /> ** [ES] In this particular case, the file was empty. He deleted the file outside VisTrails and executed it again. As he set create_file to True, he was expecting the file to be created again.<br /> ** [DK] Add random salt to hash signature when the file doesn't exist and create_file is True<br /> ** [JF: DONE] Juliana will edit the FAQ to note that the signatures are dependent on modification times<br /> <br /> * Claurissa will demo different ways to visualize the version tree<br /> ** weighting on various criteria for displaying nodes (importance from session, user, tags, etc.)<br /> ** other modes for viewing the versions (lists or timelines)<br /> ** often see only linear trees, teaching or thinking other than tree?<br /> <br /> <br /> <br /> * Upgrade issues (Dan)<br /> ** Vanilla upgrades should be in today's commit<br /> ** Added code so when doing upgrade to check whether the latest code is upgraded<br /> ** Two types of upgrades: (1) User changes the subworkflow, (2) Modules are out-of-date and need to be upgraded.<br /> ** Add higher-level features like "Upgrade All" for (1), "Import Changes to My Subworkflows", etc.<br /> ** Want a 1-1 mapping between unique subworkflow id and a subworkflow file (even if we copy the same subworkflow to different vistrails)<br /> <br /> * Offscreen bug (update from Huy)<br /> ** Mac issue when switching to Cocoa<br /> ** Huy added a QWidget for each of the render windows that is created, so should be fixed.<br /> <br /> * Gesture support<br /> ** Someone on Windows check the pinch gesture support; it doesn't work for Matthias under Parallels<br /> ** VTKCell is working now<br /> ** Cell dependent, panning now maps to middle button, pinch on the base widget<br /> ** What are Matthias's requirements here? Does touch gestures in the cells matter or only in the vt/wf views<br /> <br /> * Thumbnails: [JF: this needs to go to trac, for 2.0?] [ES: 2 tickets were created for 2.0]<br /> ** Resolution? -- [ES] problem is the size of the vistrail, but this can be configured<br /> ** Can we export to PDF instead of PNG? [ES] cannot display the thumbnail as a PDF in Qt?<br /> ** Can we have the option to save higher-res or PDF versions of the thumbnails<br /> ** Can we have preferences that allow users to save a higher-res or PDF version, maybe a checkbox<br /> ** Maybe associate a high-res version with a spreadsheet cell so that we can save a high-res version on demand<br /> ** high-res version associated with a version but allow user to initiate action from spreadsheet<br /> ** can we have thumbnails dependent on camera position [ES] doesn't like, user won't see exactly that image upon re-executing<br /> ** compact vistrail option by removing thumbnails<br /> <br /> * Tommy has rewritten the Web services module as well as updated all of the examples<br /> ** Server for one of the examples seems to be down<br /> ** Need to update the manual<br /> <br /> * [[Improve error handling/reporting]] (Tommy)<br /> ** Maybe display the intialization messages with the splash screen?<br /> ** Debugging levels have no way to access debug level<br /> ** Tommy takes first cut at trying to determine how to map print statements, notes those he is unsure of<br /> ** Developer guidelines for debug usage, use them instead of print statements in the futur<br /> <br /> * Merge functionality (Tommy) [JF: Cool!]<br /> ** Can now merge an existing vistrail into your own vistrail using menu item<br /> <br /> * Improvements to VisTrails server communication with clients to better handle errors/crashes (Phillip)<br /> ** [PM] I haven't had a chance to implement anything yet.<br /> <br /> * Data Publishing with crowdLabs. (Phillip) <br /> ** Do we want data publishing capabilities in VisTrails and crowdLabs? '''YES!!!'''<br /> ** If so...<br /> *** Currently limited data provenance for data. HTTPFile module lacks meta data, RepoSync module has poor usability.<br /> *** Persistent Data works for experimental data. Can we use it for publishing data?<br /> **** Publishing data or storing data on the server and pushing back and forth?<br /> **** What is the current scheme for identifying data on the server?<br /> **** Do we need to do better than the ids, versions, hashes that the peristence package provides?<br /> *** Do we want to utilize data publishing software like [http://thedata.org/home The Dataverse Network] or maybe [http://www.fedora-commons.org/ Fedora Commons]?<br /> ** [ES] Feature request: Can RepoSync support a directory of files?<br /> *** [PM] It can't, and I'm not sure how add it. I also don't really like the current RepoSync interface. It's not obvious how your data is being handled, hence me bring up data publishing, so hopefully we can come up with something more usable/robust.<br /> <br /> === Nov 3, 2010 ===<br /> <br /> * Upgrading subworkflows (Dan)<br /> ** add updated versions to subworkflow vistrail and push to registry<br /> ** fixed bug with version/descriptor redirects<br /> ** key remaining task is to replace the box representing the subworkflow in the top-level workflow to reflect the change to the upgraded version of the subworkflow<br /> ** many corner cases, but we should get the vanilla upgrade (working) to git<br /> ** need to ensure that each subworkflow file has its own uuid; if a subworkflow exists in two different vistrails, it should have a different uuid (namespace).<br /> ** otherwise, we can get crosstalk where one vt can update the subworkflow from another vt (we can propgate changes to other vts via merges, but this should be a user choice...)<br /> ** enhancements:<br /> *** allow users to merge subworkflows from a file into their own subworkflow (to incorporate outside changes)<br /> *** latest version is the most recent non-upgraded version; if that version has an upgrade, use the upgrade<br /> ***'''Next step: work on usability for controlflow package; explore Dave's idea of an assistant'''<br /> <br /> * Offscreen bug<br /> ** Huy is looking into this<br /> <br /> * Crash due to issues in loaded user packages (Tommy has fixed this)<br /> <br /> * Web services (Tommy)<br /> ** Package is completed, and all but one of our examples are working<br /> ** '''Tommy will test if the Web services will work when there is a proxy'''<br /> <br /> * Improve error handling/reporting (Tommy)<br /> ** we should have a single point for all error messages to pass through that is linked to both core.debug and GUI elements that display error messages. <br /> ** the goal is to implement a function that will be the single point for error messages, and that work without emitting GUI signals in core. As Huy suggested, we should have a GUI-wrapper which will be a no-op when the GUI is not instantiated---this will get rid of some of the pyqt dependencies (at least for the error messages).<br /> <br /> * Improvements to VisTrails server communication with clients to better handle errors/crashes (Phillip)<br /> ** Currently the VisTrails servers consists of: a single-threaded instance with GUI; multiple GUI-less threads<br /> ** We will extend the server API to allow clients to check the server status. The server will provide a separate socket and the client will be able to 'ping' the server; if the server is working properly, it will respond; if it does not respond, the client will have a time out<br /> ** The API will also support the ability to kill both the GUI-less threads and the single-threaded GUI instance. We can then restart the server components using Emanuele's new script.<br /> ** We should also allow users to set a per-workflow time-out; if the workflow execution takes longer than the pre-defined threshold, VisTrails will abort the execution<br /> <br /> <br /> * Update on fixing VTK Package (Wendel)<br /> <br /> ** It seems that the ParaView package has the same issues.<br /> ** '''Wendel has already checked in the new wrapper into the trunk; Emanuele will test it'''<br /> <br /> * Synchronizing VisTrails and ALPS releases (Emanuele)<br /> ** VisTrails latex package was extended so that python is no longer required. If python is not present in the system, instead of issuing a request to execute the workflow and retrieve the image from the Web, a local, previously saved image will be used. It is now also possible to embed the images of workflows into the latex file. Documentation about these and other features are currently in the README file and examples (example.tex) provided with the latex package.<br /> ** Windows and Mac beta binaries already include ALPS<br /> ** Are we distributing ALPS as part of VisTrails then? This seems backwards. ALPS should VisTrails as part of their stuff, no?<br /> *** '''We will have separate binary distributions: one with and one without ALPS'''<br /> <br /> * Is a multi-touch interface to VisTrails possible now that it's supported by Qt? (Phillip)<br /> ** Yes, Qt 4.6+ supports multi-tocuh --- '''Huy will look into this; the goal is to make this work both for the workflow builder/tree view and the spreadsheet'''<br /> ** We still need to figure out what makes sense for multi-touch, however, do we have requirements from ETH-Zurich?<br /> *** We want to map the zoom, pan, click, etc to multi-touch, so that VisTrails can be intractable on a multi-touch screen without a mouse<br /> <br /> * What is the status of the Vismashup i{Phone|Pad|Touch} app? (Phillip)<br /> ** If you mean mac binary, there's an alpha version here: http://www.sci.utah.edu/~emanuele/files/vismashup <br /> ** Do you mean iphone app? [Phillip: yes]<br /> ** '''Wendel will look into this'''<br /> *** One of the issues is how to effectively handle images that are larger than the memory on the iPods and iPads---we need to process these on the server or use a model that allows the image to be manipulated on the client<br /> *** We also need to connect the app with the crowdlabs server, so that the app can get the list of mashups<br /> *** Phillip will investigate the feasibility of working directly with JavaScript and bypass flash<br /> <br /> <br /> <br /> === Oct 19, 2010 ===<br /> <br /> * Update on Trac and Roadmap (Emanuele and David)<br /> ** Trac is now linked to git repository<br /> ** Roadmap on Trac has been cleaned up so we can hide completed milestones<br /> ** Tickets are being reassigned/revisited. Many tickets (24) are not associated with a milestone yet https://vistrails.sci.utah.edu/report/3<br /> * Synchronizing VisTrails and ALPS releases<br /> * Web services support (Tommy)<br /> ** discuss the interaction between the Web Services package and upgrades<br /> * Subworkflows update (Daniel)<br /> * Testing of new module drawing (Erik)<br /> * Aliases<br /> ** Currently, an alias is stored on a parameter. We need aliases to be stored at a higher level so that changing an alias is not a change to a parameter. The one issue is that there is a link between parmaeters and aliases in that an alias can only exist for versions that have the specified parameter. We might also have two versions where the alias points to different parameters. We could just store aliases as a root-level workflow element so that the set of aliases is versioned corrected.<br /> ** In the current implementation, the aliases parsed from the parameters are stored in a dictionary in the workflow, so storing aliases as a root-level workflows element might be the way to go --[[User:Emanuele|Emanuele]] 19:50, 15 October 2010 (MDT)<br /> * Error logging<br /> ** Suggest that we have a single point for all error messages to pass through that is linked to both core.debug and GUI elements that display error messages. This should improve our error handling significantly<br /> * PyQt dependencies in core<br /> ** Can we get rid of PyQt dependencies in core (and db)? This would probably require moving the signals/slots that we currently use to a similar Python implementation which shouldn't be too difficult. This would also require the specification of configuration widgets not as classes but rather some text that can be used to import the GUI elements only when using the GUI code. However, what would happen with workflows that run in command-line mode but require some graphical output?<br /> * Ports<br /> ** One annoying part of the current VisTrails model is that users are not given any visual indication that a port value is already set by a function or vice versa by a connection. In addition, we have no way to specify or enforce cardinality on ports at design time. It seems like we could allow developers to specify when a port should not be connected to more than one value (via a function or a connection), and give some visual feedback when a port has already been specified in one manner.<br /> <br /> ==== Meeting Notes ====<br /> <br /> * Bug regarding userpackages at initial vistrails startup causing startup to fail.<br /> ** Still an issue, ticket still open as a 1.6 milestone<br /> * Copy-paste bug<br /> ** Shortcut not working on Mac. Focus makes shortcut try to copy pipeline.<br /> ** Current fix appends to clipboard as users type - this is not a good fix.<br /> * Synchronizing ALPS and VisTrails releases<br /> ** 1.6 scheduled for Dec. 1<br /> * Webservices<br /> ** New library being used, but it's very low level.<br /> ** Simple types may cause problems with some web-services using complex or XML-based types.<br /> ** Need to make sure that this library can handle at least MOST of the web-services out there.<br /> * Sub-workflows<br /> ** Daniel's changes seem functional and very slick.<br /> ** A little more work on it seems necessary.<br /> ** Need some easy GUI mechanism to delete a subworkflow.<br /> ** Is the exclamation point in the module draw the best way to handle this? Make a tool-tip to explain it.<br /> * VTK Wrapping<br /> ** Changes to VTK Python wrapping is causing headaches - particularly in backwards compatibility.<br /> ** Method typing has changed to expect things like "List" or "Vector" - makes things hard on the user when defining these inputs.<br /> ** Removing List and Vector types and replacing them with dynamically generated versions from Tuples.<br /> * Aliases - Agenda pushed until after Vis.<br /> * Error Logging - Unify some error handling to improve how exceptions are dealt with. More on this after Vis.<br /> * Refactoring out PyQt dependencies from Core/Db/etc - Need to move all the signals/slots into GUI elements. Questions come when a workflow uses GUI elements as inputs. - More after vis.<br /> * Ports - We need a way of showing how a port is specified - function vs. input port. Cardinality must be established in these cases. Ordering on multiple connections should be handled in some way - whether it's just a documentation issue or a more fundamental one.<br /> ** Should we have a tutorial mode that bring up tooltips when someone does something new as if it were a live tutorial.<br /> <br /> <br /> === Oct 12, 2010 ===<br /> <br /> * Trac and Roadmap (Emanuele): Emanuele suggested we should create a development roadmap and make it a permanent item on the agenda. The idea is to go through the Trac tickets and use them to build the roadmap that would be made ''public''.<br /> ** go through trac and sort by priority<br /> ** add other suggestions to roadmap as well<br /> ** admins can add milestones via the admin tab<br /> ** ''David will assign tasks on list; those assigned need to follow up and check and fix them''<br /> <br /> * Update on [[PythonSource error reporting and logging]] (Tommy)<br /> ** ''Need to make sure that all messages are printed through debug.X, so that the debug level (verboseness in preferences) is correctly used''<br /> ** Can cut/copy/paste now<br /> ** Can we prevent the delete from happening?<br /> ** Phillip noted the spyder project has an interactive python console: http://packages.python.org/spyder/<br /> ** Can press enter to get to the end<br /> ** Need to check if this works or if we need to change<br /> ** Now have ability to see stack trace from the GUI (triangle menu)<br /> ** ''Suggest using a dialog to display trace when this item is selected instead of printing to console''<br /> ** Also saving stack trace to the execution log<br /> ** Also printing debug information to vistrails log<br /> ** ''Add trac item to fix prints to use debug''<br /> ** ''Try to fix core.debug to have gui.debug to eliminate Qt dependency''<br /> <br /> * Update on subworkflow (Daniel)<br /> ** Notification when subworkflow is outdated triggered<br /> ** new_abstraction signal from registry in addition to new_module signal so that we can check if we need subworkflows to update<br /> ** How to display the possible upgrades (can be upgraded, can be upgraded but may break, etc.)<br /> <br /> * New utility added by Carlos (currently at scripts/module_appearance) to draw fringes visually, which generates output that can be pasted directly into the add_module call. <br /> ** ''Erik will test this on Mac and Windows.''<br /> <br /> * Preparing a "Get Started" tutorial for SIGMOD repeatability <br /> ** Do we have instructions on how to use the latex package without crowdlabs?<br /> *** links to actual workflows in latex<br /> *** have README and example for latex in the source<br /> ** Metadata associated with VisTrails: should we have the ability to add vistrail-level metadata? e.g., who created the vistrail, its purpose, etc.<br /> *** have pointers to paper to workflow and vistrails<br /> *** add pointers to papers<br /> *** allow access to add/edit annotations at the vistrail level<br /> *** ''add GUI element to allow people to edit/add annotations''<br /> <br /> * Maintaining VTK package<br /> ** have four classes that don't wrap, have error even in python level<br /> ** works well with VisTrails otherwise<br /> ** haven't checked the new changes from the wrapping<br /> ** ''Wendel will check on the new version''<br /> <br /> * [[Web services package]] (Tommy)<br /> <br /> === Oct 5, 2010 ===<br /> <br /> * Welcome Tommy!<br /> <br /> * Update on subworkflow (Daniel)<br /> <br /> * Issues raised by Matthias:<br /> <br /> - need to automatically upgrade subworkflows. Currently it is necessary to manually upgrade a subworkflow whenever the version of one of the modules inside them changes.<br /> <br /> - explorations do not detect an MplFigureCell embedded in a subworkflow.</div>

Mates https://www.vistrails.org//index.php?title=Development&diff=3723 Development 2010-12-02T14:41:47Z

<p>Mates: /* Weekly Meetings */</p> <hr /> <div>== Roadmap ==<br /> <br /> <br /> == Weekly Meetings ==<br /> <br /> === Dec 7, 2010 ===<br /> <br /> * On merging master and crowdlabs git branches<br /> ** some updates on the crowdlabs branch need to be merged/patched into the master branch<br /> <br /> * PythonSource editor<br /> ** modal or multiple window editors?<br /> <br /> === Nov 30, 2010 ===<br /> <br /> * Subworkflow Upgrades (Daniel)<br /> ** Should be completed finished tonight <br /> ** Local subworkflow upgrades should be working now<br /> ** Need to test these after the checkin<br /> ** Using a dummy controller<br /> <br /> * Assistant for control package (Daniel)<br /> ** Look at examples<br /> ** Try to make an assistant to modify workflow using currently selected modules as looping group<br /> <br /> * [[Improve error handling/reporting]]<br /> ** Tons of message boxes pop up when a workflow has a bunch of errors, UI response is bad after this...<br /> ** Update existing error messages<br /> ** Show Details rendering on Mac<br /> ** What to do when multiple errors occur in a single event?<br /> ** New dialog that shows "next message" but allows user to dismiss messages or view all in the messages window<br /> ** Color coding for messages: use gui/theme.py to configure: maybe try black for warning, gray for log level messages<br /> <br /> * Data and workflow browser (Nivan will give a status update)<br /> <br /> * Saving Explorations (Feature Request from ALPS)<br /> ** We already save parameter explorations with the workflow in the version tree. As is, we save only the last exploration that was done for a given workflow. If you ship the vistrail, the saved parameter explorations will be there. Is this sufficient?<br /> ** [MT] Yes, that takes us a long way. Could one add a button in the explorations to save a certain set of explorations as a new version, like you do with the camera angle in the spreadsheet? That way one could store more than one set of exploration parameters without forcing a new version to be created with every change?<br /> ** Eventually we should provide a selector in parameter exploration<br /> ** Fastest to just allow a null action new version (a bit clunky) but wouldn't require too many changes<br /> <br /> * Clean up/prune persistent directory<br /> ** What are the concrete use cases for this?<br /> ** [MT] Here are two concrete use cases that appeared over the last week:<br /> *** a project is done, the important results copied to an archive and I want to get rid of all its intermediate files, etc., to save space - but not those files associated with other projects. Or, similarly, I'm done testing some workflows, want to remove all (partially buggy or irrelevant) files, and then start with production runs.<br /> *** there was a problem in a workflow(e.g. a bug in HTTPFile module with binary files or a bug in some user code). I want to force a certain persisted directory to be "forgotten", so that I can repair the wrong files. Bumping the version number of the ALPS package would be one way, but that is overkill since it would immediately invalidate all persisted files. Removing the whole persistence directory is just as radical.<br /> ** just testing simple tutorials with small files leaves me with 2 GB of persistent data after 10 days, in 142 persistent directories or files. I need some way to clean this up that is less radical then removing all persistent directories and files. This could be done, by e.g. removing all those which do not occur in tagged or leaf workflows in a certain set of vistrails. Or marking those which occur in certain vistrails with certain labels, and then being able to prune all those with or without some label/tag. Another option might be having the ability to choose from various repositories.<br /> ** [DK] follow up on this<br /> <br /> * Improvements to VisTrails server communication with clients to better handle errors/crashes (Phillip)<br /> ** work in progress<br /> <br /> * Data Publishing with crowdLabs. (Phillip) <br /> ** Do we want data publishing capabilities in VisTrails and crowdLabs? '''YES!!!'''<br /> ** If so...<br /> *** Currently limited data provenance for data. HTTPFile module lacks meta data, RepoSync module has poor usability.<br /> *** Persistent Data works for experimental data. Can we use it for publishing data?<br /> **** Publishing data or storing data on the server and pushing back and forth?<br /> **** What is the current scheme for identifying data on the server?<br /> **** Do we need to do better than the ids, versions, hashes that the peristence package provides?<br /> *** Do we want to utilize data publishing software like [http://thedata.org/home The Dataverse Network] or maybe [http://www.fedora-commons.org/ Fedora Commons]?<br /> ** [ES] Feature request: Can RepoSync support a directory of files?<br /> *** [PM] It can't, and I'm not sure how add it. I also don't really like the current RepoSync interface. It's not obvious how your data is being handled, hence me bring up data publishing, so hopefully we can come up with something more usable/robust.<br /> ** persistence used more for exploratory<br /> ** want to focus on more archive<br /> ** move from exploratory data to published<br /> ** Dataverse: essentially a site to curate data, archive data, host their own site, each university has their own server, linked together<br /> ** persistent store integration: maybe use git to do this, can add to local repositories to move to git<br /> ** use some of the dataverse: standard hash schemes, etc.<br /> ** privacy: can users push workflows without exposing data<br /> ** permissions for who can check things out: download permissions, upload permissions<br /> ** look into setting up persistent store on <br /> <br /> * DisplayWall (Wendel)<br /> ** new version of vistrails receives rotate/scale messages much more slowly<br /> ** using the vtk version that is already there<br /> ** comparing VisTrails 1.2 with nightly src<br /> ** test when the messages are received on the clients<br /> ** [ES] Changed something in the VTKCell, using the Qt interactor from vtk now, just uncomment this (vtkcell.py file)<br /> ** some commands are missed; check why this is happening<br /> ** preferences switch? 2x2 not 2x3<br /> ** check what modifications exist in the client code that's not exactly 1.2?<br /> ** new version of vtk package changed some things<br /> ** update vtk upgrade: resetcamera and addrgbpoint => _# versions<br /> ** Demos on displaywall: Wendel has to keep both VisTrails running and stop work when visitors stop by unannounced<br /> <br /> * Relative paths in workflows, how to deal with them? (Emanuele)<br /> ** [ES] Updated all examples to use HTTPFile for these<br /> ** some of these had to change completely (color widget), kept tree the same format<br /> <br /> === Nov 16, 2010 ===<br /> <br /> * Data and workflow browser (Nivan)<br /> ** Check into git branch (Dave)<br /> ** Add data support to the vistrails, workflows, and execs already exposed in the browser<br /> ** Add thumbnails to the browser<br /> ** Bittorrent support?<br /> <br /> * Upgrade issues -- a wrap? (Daniel)<br /> ** Fixing some issues with namespaces<br /> ** Upgrades for subworkflows that are included with packages, need to save these upgrades with the vt file as a "local abstraction" to ensure provenance. May be able to tie these back to the package versions once the package is upgraded.<br /> <br /> * Test Suite<br /> ** Matthias has his own test suite now<br /> ** We should make sure that our tests are up-to-date and run them before releases<br /> <br /> * Web services on users' guide (Tommy)<br /> ** Users' guide needs only a few changes. One of the changes should be a note making it explicit that we are using a new library and the old one was deprecated.<br /> ** Do we want to update the 39-page guide as well? For now, just update the users' guide. Don't distribute this guide. Just include it in a zip file that includes the old version.<br /> <br /> * [[Improve error handling/reporting]] (Tommy)<br /> ** print statements, has a new window<br /> ** add a menu item to view message window and also keep all messages from the session<br /> ** look into writing QMessageBox errors and warning to message window<br /> ** write all messages to a single window, write critical messages in red?<br /> ** have a utility method that encapsulates the gui error display and the core.debug display (if not running gui, only write to core.debug)<br /> ** don't display stack trace in a qmessagebox? -- try to emulate the invalid pipeline with "Show Details" button.<br /> ** make it easier for users to report errors by having a "copy" or "send to developers" button to report the details of an error.<br /> <br /> * DisplayWall (Wendel)<br /> ** What examples should be in the SVN? If we decide to keep some examples in the svn, some of them use data that should be along.<br /> *** Indicate progress of HTTPFile<br /> *** Had to upgrade examples to work with new examples of VisTrails<br /> *** VisTrails can send progress information with another thread like crowdlabs can do<br /> *** Each machine has a .vistrails and needs to download own version of data, make a single .vistrails for all machines?<br /> *** All slave machines can use the server as a cache?<br /> ** DisplayWall Client on the Apple Store? I dont like the idea of sharing the client source code. And putting the app without login/password will allow anyone to access our displaywall.<br /> *** Add a password for this<br /> *** Do user management?<br /> <br /> * VTK package:<br /> ** Have two ResetCamera modules now<br /> ** Need to write upgrade code<br /> ** Only with vtk 5.7 right now<br /> ** Encode library version in the package version?<br /> ** For reproducibility, we probably want to stick to specific versions<br /> ** We still need some ability to run new versions of vtk for testing, new features, etc., too...<br /> <br /> * Relative paths in workflows, how to deal with them? (Emanuele)<br /> ** Should we use dataDirectory?<br /> ** Change examples to not use hard-coded files?<br /> ** Problem with workflows, not data files<br /> ** On Windows, use ../examples<br /> ** On Mac, use ../../../examples<br /> ** dataDirectory not used, I think<br /> ** create "$VT_EXAMPLES"?<br /> ** problem is using new versions of VisTrails, data may change so referencing old location will fail<br /> ** don't allow users to change this variable<br /> ** convert to use HTTPFile<br /> <br /> * Data files and the GIT Repository (Emanuele)<br /> ** make a separate repository or use the submodule git functionality<br /> ** go with submodule<br /> <br /> * Next release: What still needs to be done?<br /> ** high priority from ALPS list<br /> ** Dan's fixes to the upgrade workflow<br /> ** Date for release? Official release in beginning of December for ALPS<br /> ** All developers should run test suite on push<br /> ** Could make this somewhat automatic, run with nightly release script and email -dev list<br /> ** Could have different classes of tests, make sure that critical ones pass<br /> ** Tests have a bunch of errors currently<br /> <br /> * Saving explorations:<br /> ** Keeps track of the latest exploration for each version<br /> ** Check with Matthias what the requirements for this are<br /> ** Have way to send just specification to a server (run this workflow with this parameter exploration)?<br /> ** Can we have an xml spec that specifies the parameter exploration (export parameter exploration)<br /> ** Currently persisting via xml and then converting to string so we already have an xml serialization<br /> ** Dan wrote API that takes xml string and populates the param_exp gui with the appropriate values<br /> ** Need to back the GUI state with core state to enable short-circuiting the gui<br /> <br /> <br /> * Box input ports should be colored (e.g. black) if they are already set by an internal method. Another color (e.g. red) could be used to indicate mandatory input ports.<br /> ** Currently, non-optional ports are square, shown. Optionals are circles and must be enabled<br /> ** Add a third category for mandatory, problem is backward compat.<br /> ** [ES] Colors are already overloaded, not for ports, use shading instead?<br /> <br /> === Nov 9, 2010 ===<br /> <br /> * FAQ<br /> ** As we reply to users queries, let's add the question and answer to the FAQ!<br /> ** Just a reminder<br /> <br /> * Caching of File module<br /> ** There was a message from a user who got confused because this module is cached by default (and silently). And in his application, since the file actually changed in between runs, he did not see the 'expected' result.<br /> ** Should we keep a hash for files and check whether they changed?<br /> ** Should we no cache File by default?<br /> ** [DK] File is supposed to have a special signature computation that detects changes in the file contents so this may be a bug instead. It does work, it seems to be an issue with the create_file<br /> ** [ES] In this particular case, the file was empty. He deleted the file outside VisTrails and executed it again. As he set create_file to True, he was expecting the file to be created again.<br /> ** [DK] Add random salt to hash signature when the file doesn't exist and create_file is True<br /> ** [JF: DONE] Juliana will edit the FAQ to note that the signatures are dependent on modification times<br /> <br /> * Claurissa will demo different ways to visualize the version tree<br /> ** weighting on various criteria for displaying nodes (importance from session, user, tags, etc.)<br /> ** other modes for viewing the versions (lists or timelines)<br /> ** often see only linear trees, teaching or thinking other than tree?<br /> <br /> <br /> <br /> * Upgrade issues (Dan)<br /> ** Vanilla upgrades should be in today's commit<br /> ** Added code so when doing upgrade to check whether the latest code is upgraded<br /> ** Two types of upgrades: (1) User changes the subworkflow, (2) Modules are out-of-date and need to be upgraded.<br /> ** Add higher-level features like "Upgrade All" for (1), "Import Changes to My Subworkflows", etc.<br /> ** Want a 1-1 mapping between unique subworkflow id and a subworkflow file (even if we copy the same subworkflow to different vistrails)<br /> <br /> * Offscreen bug (update from Huy)<br /> ** Mac issue when switching to Cocoa<br /> ** Huy added a QWidget for each of the render windows that is created, so should be fixed.<br /> <br /> * Gesture support<br /> ** Someone on Windows check the pinch gesture support; it doesn't work for Matthias under Parallels<br /> ** VTKCell is working now<br /> ** Cell dependent, panning now maps to middle button, pinch on the base widget<br /> ** What are Matthias's requirements here? Does touch gestures in the cells matter or only in the vt/wf views<br /> <br /> * Thumbnails: [JF: this needs to go to trac, for 2.0?] [ES: 2 tickets were created for 2.0]<br /> ** Resolution? -- [ES] problem is the size of the vistrail, but this can be configured<br /> ** Can we export to PDF instead of PNG? [ES] cannot display the thumbnail as a PDF in Qt?<br /> ** Can we have the option to save higher-res or PDF versions of the thumbnails<br /> ** Can we have preferences that allow users to save a higher-res or PDF version, maybe a checkbox<br /> ** Maybe associate a high-res version with a spreadsheet cell so that we can save a high-res version on demand<br /> ** high-res version associated with a version but allow user to initiate action from spreadsheet<br /> ** can we have thumbnails dependent on camera position [ES] doesn't like, user won't see exactly that image upon re-executing<br /> ** compact vistrail option by removing thumbnails<br /> <br /> * Tommy has rewritten the Web services module as well as updated all of the examples<br /> ** Server for one of the examples seems to be down<br /> ** Need to update the manual<br /> <br /> * [[Improve error handling/reporting]] (Tommy)<br /> ** Maybe display the intialization messages with the splash screen?<br /> ** Debugging levels have no way to access debug level<br /> ** Tommy takes first cut at trying to determine how to map print statements, notes those he is unsure of<br /> ** Developer guidelines for debug usage, use them instead of print statements in the futur<br /> <br /> * Merge functionality (Tommy) [JF: Cool!]<br /> ** Can now merge an existing vistrail into your own vistrail using menu item<br /> <br /> * Improvements to VisTrails server communication with clients to better handle errors/crashes (Phillip)<br /> ** [PM] I haven't had a chance to implement anything yet.<br /> <br /> * Data Publishing with crowdLabs. (Phillip) <br /> ** Do we want data publishing capabilities in VisTrails and crowdLabs? '''YES!!!'''<br /> ** If so...<br /> *** Currently limited data provenance for data. HTTPFile module lacks meta data, RepoSync module has poor usability.<br /> *** Persistent Data works for experimental data. Can we use it for publishing data?<br /> **** Publishing data or storing data on the server and pushing back and forth?<br /> **** What is the current scheme for identifying data on the server?<br /> **** Do we need to do better than the ids, versions, hashes that the peristence package provides?<br /> *** Do we want to utilize data publishing software like [http://thedata.org/home The Dataverse Network] or maybe [http://www.fedora-commons.org/ Fedora Commons]?<br /> ** [ES] Feature request: Can RepoSync support a directory of files?<br /> *** [PM] It can't, and I'm not sure how add it. I also don't really like the current RepoSync interface. It's not obvious how your data is being handled, hence me bring up data publishing, so hopefully we can come up with something more usable/robust.<br /> <br /> === Nov 3, 2010 ===<br /> <br /> * Upgrading subworkflows (Dan)<br /> ** add updated versions to subworkflow vistrail and push to registry<br /> ** fixed bug with version/descriptor redirects<br /> ** key remaining task is to replace the box representing the subworkflow in the top-level workflow to reflect the change to the upgraded version of the subworkflow<br /> ** many corner cases, but we should get the vanilla upgrade (working) to git<br /> ** need to ensure that each subworkflow file has its own uuid; if a subworkflow exists in two different vistrails, it should have a different uuid (namespace).<br /> ** otherwise, we can get crosstalk where one vt can update the subworkflow from another vt (we can propgate changes to other vts via merges, but this should be a user choice...)<br /> ** enhancements:<br /> *** allow users to merge subworkflows from a file into their own subworkflow (to incorporate outside changes)<br /> *** latest version is the most recent non-upgraded version; if that version has an upgrade, use the upgrade<br /> ***'''Next step: work on usability for controlflow package; explore Dave's idea of an assistant'''<br /> <br /> * Offscreen bug<br /> ** Huy is looking into this<br /> <br /> * Crash due to issues in loaded user packages (Tommy has fixed this)<br /> <br /> * Web services (Tommy)<br /> ** Package is completed, and all but one of our examples are working<br /> ** '''Tommy will test if the Web services will work when there is a proxy'''<br /> <br /> * Improve error handling/reporting (Tommy)<br /> ** we should have a single point for all error messages to pass through that is linked to both core.debug and GUI elements that display error messages. <br /> ** the goal is to implement a function that will be the single point for error messages, and that work without emitting GUI signals in core. As Huy suggested, we should have a GUI-wrapper which will be a no-op when the GUI is not instantiated---this will get rid of some of the pyqt dependencies (at least for the error messages).<br /> <br /> * Improvements to VisTrails server communication with clients to better handle errors/crashes (Phillip)<br /> ** Currently the VisTrails servers consists of: a single-threaded instance with GUI; multiple GUI-less threads<br /> ** We will extend the server API to allow clients to check the server status. The server will provide a separate socket and the client will be able to 'ping' the server; if the server is working properly, it will respond; if it does not respond, the client will have a time out<br /> ** The API will also support the ability to kill both the GUI-less threads and the single-threaded GUI instance. We can then restart the server components using Emanuele's new script.<br /> ** We should also allow users to set a per-workflow time-out; if the workflow execution takes longer than the pre-defined threshold, VisTrails will abort the execution<br /> <br /> <br /> * Update on fixing VTK Package (Wendel)<br /> <br /> ** It seems that the ParaView package has the same issues.<br /> ** '''Wendel has already checked in the new wrapper into the trunk; Emanuele will test it'''<br /> <br /> * Synchronizing VisTrails and ALPS releases (Emanuele)<br /> ** VisTrails latex package was extended so that python is no longer required. If python is not present in the system, instead of issuing a request to execute the workflow and retrieve the image from the Web, a local, previously saved image will be used. It is now also possible to embed the images of workflows into the latex file. Documentation about these and other features are currently in the README file and examples (example.tex) provided with the latex package.<br /> ** Windows and Mac beta binaries already include ALPS<br /> ** Are we distributing ALPS as part of VisTrails then? This seems backwards. ALPS should VisTrails as part of their stuff, no?<br /> *** '''We will have separate binary distributions: one with and one without ALPS'''<br /> <br /> * Is a multi-touch interface to VisTrails possible now that it's supported by Qt? (Phillip)<br /> ** Yes, Qt 4.6+ supports multi-tocuh --- '''Huy will look into this; the goal is to make this work both for the workflow builder/tree view and the spreadsheet'''<br /> ** We still need to figure out what makes sense for multi-touch, however, do we have requirements from ETH-Zurich?<br /> *** We want to map the zoom, pan, click, etc to multi-touch, so that VisTrails can be intractable on a multi-touch screen without a mouse<br /> <br /> * What is the status of the Vismashup i{Phone|Pad|Touch} app? (Phillip)<br /> ** If you mean mac binary, there's an alpha version here: http://www.sci.utah.edu/~emanuele/files/vismashup <br /> ** Do you mean iphone app? [Phillip: yes]<br /> ** '''Wendel will look into this'''<br /> *** One of the issues is how to effectively handle images that are larger than the memory on the iPods and iPads---we need to process these on the server or use a model that allows the image to be manipulated on the client<br /> *** We also need to connect the app with the crowdlabs server, so that the app can get the list of mashups<br /> *** Phillip will investigate the feasibility of working directly with JavaScript and bypass flash<br /> <br /> <br /> <br /> === Oct 19, 2010 ===<br /> <br /> * Update on Trac and Roadmap (Emanuele and David)<br /> ** Trac is now linked to git repository<br /> ** Roadmap on Trac has been cleaned up so we can hide completed milestones<br /> ** Tickets are being reassigned/revisited. Many tickets (24) are not associated with a milestone yet https://vistrails.sci.utah.edu/report/3<br /> * Synchronizing VisTrails and ALPS releases<br /> * Web services support (Tommy)<br /> ** discuss the interaction between the Web Services package and upgrades<br /> * Subworkflows update (Daniel)<br /> * Testing of new module drawing (Erik)<br /> * Aliases<br /> ** Currently, an alias is stored on a parameter. We need aliases to be stored at a higher level so that changing an alias is not a change to a parameter. The one issue is that there is a link between parmaeters and aliases in that an alias can only exist for versions that have the specified parameter. We might also have two versions where the alias points to different parameters. We could just store aliases as a root-level workflow element so that the set of aliases is versioned corrected.<br /> ** In the current implementation, the aliases parsed from the parameters are stored in a dictionary in the workflow, so storing aliases as a root-level workflows element might be the way to go --[[User:Emanuele|Emanuele]] 19:50, 15 October 2010 (MDT)<br /> * Error logging<br /> ** Suggest that we have a single point for all error messages to pass through that is linked to both core.debug and GUI elements that display error messages. This should improve our error handling significantly<br /> * PyQt dependencies in core<br /> ** Can we get rid of PyQt dependencies in core (and db)? This would probably require moving the signals/slots that we currently use to a similar Python implementation which shouldn't be too difficult. This would also require the specification of configuration widgets not as classes but rather some text that can be used to import the GUI elements only when using the GUI code. However, what would happen with workflows that run in command-line mode but require some graphical output?<br /> * Ports<br /> ** One annoying part of the current VisTrails model is that users are not given any visual indication that a port value is already set by a function or vice versa by a connection. In addition, we have no way to specify or enforce cardinality on ports at design time. It seems like we could allow developers to specify when a port should not be connected to more than one value (via a function or a connection), and give some visual feedback when a port has already been specified in one manner.<br /> <br /> ==== Meeting Notes ====<br /> <br /> * Bug regarding userpackages at initial vistrails startup causing startup to fail.<br /> ** Still an issue, ticket still open as a 1.6 milestone<br /> * Copy-paste bug<br /> ** Shortcut not working on Mac. Focus makes shortcut try to copy pipeline.<br /> ** Current fix appends to clipboard as users type - this is not a good fix.<br /> * Synchronizing ALPS and VisTrails releases<br /> ** 1.6 scheduled for Dec. 1<br /> * Webservices<br /> ** New library being used, but it's very low level.<br /> ** Simple types may cause problems with some web-services using complex or XML-based types.<br /> ** Need to make sure that this library can handle at least MOST of the web-services out there.<br /> * Sub-workflows<br /> ** Daniel's changes seem functional and very slick.<br /> ** A little more work on it seems necessary.<br /> ** Need some easy GUI mechanism to delete a subworkflow.<br /> ** Is the exclamation point in the module draw the best way to handle this? Make a tool-tip to explain it.<br /> * VTK Wrapping<br /> ** Changes to VTK Python wrapping is causing headaches - particularly in backwards compatibility.<br /> ** Method typing has changed to expect things like "List" or "Vector" - makes things hard on the user when defining these inputs.<br /> ** Removing List and Vector types and replacing them with dynamically generated versions from Tuples.<br /> * Aliases - Agenda pushed until after Vis.<br /> * Error Logging - Unify some error handling to improve how exceptions are dealt with. More on this after Vis.<br /> * Refactoring out PyQt dependencies from Core/Db/etc - Need to move all the signals/slots into GUI elements. Questions come when a workflow uses GUI elements as inputs. - More after vis.<br /> * Ports - We need a way of showing how a port is specified - function vs. input port. Cardinality must be established in these cases. Ordering on multiple connections should be handled in some way - whether it's just a documentation issue or a more fundamental one.<br /> ** Should we have a tutorial mode that bring up tooltips when someone does something new as if it were a live tutorial.<br /> <br /> <br /> === Oct 12, 2010 ===<br /> <br /> * Trac and Roadmap (Emanuele): Emanuele suggested we should create a development roadmap and make it a permanent item on the agenda. The idea is to go through the Trac tickets and use them to build the roadmap that would be made ''public''.<br /> ** go through trac and sort by priority<br /> ** add other suggestions to roadmap as well<br /> ** admins can add milestones via the admin tab<br /> ** ''David will assign tasks on list; those assigned need to follow up and check and fix them''<br /> <br /> * Update on [[PythonSource error reporting and logging]] (Tommy)<br /> ** ''Need to make sure that all messages are printed through debug.X, so that the debug level (verboseness in preferences) is correctly used''<br /> ** Can cut/copy/paste now<br /> ** Can we prevent the delete from happening?<br /> ** Phillip noted the spyder project has an interactive python console: http://packages.python.org/spyder/<br /> ** Can press enter to get to the end<br /> ** Need to check if this works or if we need to change<br /> ** Now have ability to see stack trace from the GUI (triangle menu)<br /> ** ''Suggest using a dialog to display trace when this item is selected instead of printing to console''<br /> ** Also saving stack trace to the execution log<br /> ** Also printing debug information to vistrails log<br /> ** ''Add trac item to fix prints to use debug''<br /> ** ''Try to fix core.debug to have gui.debug to eliminate Qt dependency''<br /> <br /> * Update on subworkflow (Daniel)<br /> ** Notification when subworkflow is outdated triggered<br /> ** new_abstraction signal from registry in addition to new_module signal so that we can check if we need subworkflows to update<br /> ** How to display the possible upgrades (can be upgraded, can be upgraded but may break, etc.)<br /> <br /> * New utility added by Carlos (currently at scripts/module_appearance) to draw fringes visually, which generates output that can be pasted directly into the add_module call. <br /> ** ''Erik will test this on Mac and Windows.''<br /> <br /> * Preparing a "Get Started" tutorial for SIGMOD repeatability <br /> ** Do we have instructions on how to use the latex package without crowdlabs?<br /> *** links to actual workflows in latex<br /> *** have README and example for latex in the source<br /> ** Metadata associated with VisTrails: should we have the ability to add vistrail-level metadata? e.g., who created the vistrail, its purpose, etc.<br /> *** have pointers to paper to workflow and vistrails<br /> *** add pointers to papers<br /> *** allow access to add/edit annotations at the vistrail level<br /> *** ''add GUI element to allow people to edit/add annotations''<br /> <br /> * Maintaining VTK package<br /> ** have four classes that don't wrap, have error even in python level<br /> ** works well with VisTrails otherwise<br /> ** haven't checked the new changes from the wrapping<br /> ** ''Wendel will check on the new version''<br /> <br /> * [[Web services package]] (Tommy)<br /> <br /> === Oct 5, 2010 ===<br /> <br /> * Welcome Tommy!<br /> <br /> * Update on subworkflow (Daniel)<br /> <br /> * Issues raised by Matthias:<br /> <br /> - need to automatically upgrade subworkflows. Currently it is necessary to manually upgrade a subworkflow whenever the version of one of the modules inside them changes.<br /> <br /> - explorations do not detect an MplFigureCell embedded in a subworkflow.</div>

Mates https://www.vistrails.org//index.php?title=Development&diff=3672 Development 2010-11-09T09:09:30Z

<p>Mates: /* Nov 9, 2010 */</p> <hr /> <div>== Roadmap ==<br /> <br /> <br /> == Weekly Meetings ==<br /> <br /> === Nov 9, 2010 ===<br /> <br /> * FAQ<br /> ** As we reply to users queries, let's add the question and answer to the FAQ!<br /> <br /> * Caching of File module<br /> ** There was a message from a user who got confused because this module is cached by default (and silently). And in his application, since the file actually changed in between runs, he did not see the 'expected' result.<br /> ** Should we keep a hash for files and check whether they changed?<br /> ** Should we no cache File by default?<br /> ** [DK] File is supposed to have a special signature computation that detects changes in the file contents so this may be a bug instead.<br /> ** [ES] In this particular case, the file was empty. He deleted the file outside VisTrails and executed it again. As he set create_file to True, he was expecting the file to be created again.<br /> <br /> * Claurissa will demo different ways to visualize the version tree<br /> <br /> * Data and workflow browser (Nivan)<br /> <br /> * Upgrade issues<br /> <br /> * Offscreen bug (update from Huy)<br /> <br /> * Gesture support<br /> ** Someone on Windows check the pinch gesture support; it doesn't work for Matthias under Parallels<br /> <br /> * Improve error handling/reporting (Tommy)<br /> <br /> * Improvements to VisTrails server communication with clients to better handle errors/crashes (Phillip)<br /> ** [PM] I haven't had a chance to implement anything yet.<br /> <br /> * Data Publishing with crowdLabs. (Phillip) <br /> ** Do we want data publishing capabilities in VisTrails and crowdLabs? '''YES!!!'''<br /> ** If so...<br /> *** Currently limited data provenance for data. HTTPFile module lacks meta data, RepoSync module has poor usability.<br /> *** Persistent Data works for experimental data. Can we use it for publishing data?<br /> **** Publishing data or storing data on the server and pushing back and forth?<br /> **** What is the current scheme for identifying data on the server?<br /> **** Do we need to do better than the ids, versions, hashes that the peristence package provides?<br /> *** Do we want to utilize data publishing software like [http://thedata.org/home The Dataverse Network] or maybe [http://www.fedora-commons.org/ Fedora Commons]?<br /> ** [ES] Feature request: Can RepoSync support a directory of files?<br /> *** [PM] It can't, and I'm not sure how add it. I also don't really like the current RepoSync interface. It's not obvious how your data is being handled, hence me bring up data publishing, so hopefully we can come up with something more usable/robust.<br /> <br /> === Nov 3, 2010 ===<br /> <br /> * Upgrading subworkflows (Dan)<br /> ** add updated versions to subworkflow vistrail and push to registry<br /> ** fixed bug with version/descriptor redirects<br /> ** key remaining task is to replace the box representing the subworkflow in the top-level workflow to reflect the change to the upgraded version of the subworkflow<br /> ** many corner cases, but we should get the vanilla upgrade (working) to git<br /> ** need to ensure that each subworkflow file has its own uuid; if a subworkflow exists in two different vistrails, it should have a different uuid (namespace).<br /> ** otherwise, we can get crosstalk where one vt can update the subworkflow from another vt (we can propgate changes to other vts via merges, but this should be a user choice...)<br /> ** enhancements:<br /> *** allow users to merge subworkflows from a file into their own subworkflow (to incorporate outside changes)<br /> *** latest version is the most recent non-upgraded version; if that version has an upgrade, use the upgrade<br /> ***'''Next step: work on usability for controlflow package; explore Dave's idea of an assistant'''<br /> <br /> * Offscreen bug<br /> ** Huy is looking into this<br /> <br /> * Crash due to issues in loaded user packages (Tommy has fixed this)<br /> <br /> * Web services (Tommy)<br /> ** Package is completed, and all but one of our examples are working<br /> ** '''Tommy will test if the Web services will work when there is a proxy'''<br /> <br /> * Improve error handling/reporting (Tommy)<br /> ** we should have a single point for all error messages to pass through that is linked to both core.debug and GUI elements that display error messages. <br /> ** the goal is to implement a function that will be the single point for error messages, and that work without emitting GUI signals in core. As Huy suggested, we should have a GUI-wrapper which will be a no-op when the GUI is not instantiated---this will get rid of some of the pyqt dependencies (at least for the error messages).<br /> <br /> * Improvements to VisTrails server communication with clients to better handle errors/crashes (Phillip)<br /> ** Currently the VisTrails servers consists of: a single-threaded instance with GUI; multiple GUI-less threads<br /> ** We will extend the server API to allow clients to check the server status. The server will provide a separate socket and the client will be able to 'ping' the server; if the server is working properly, it will respond; if it does not respond, the client will have a time out<br /> ** The API will also support the ability to kill both the GUI-less threads and the single-threaded GUI instance. We can then restart the server components using Emanuele's new script.<br /> ** We should also allow users to set a per-workflow time-out; if the workflow execution takes longer than the pre-defined threshold, VisTrails will abort the execution<br /> <br /> <br /> * Update on fixing VTK Package (Wendel)<br /> <br /> ** It seems that the ParaView package has the same issues.<br /> ** '''Wendel has already checked in the new wrapper into the trunk; Emanuele will test it'''<br /> <br /> * Synchronizing VisTrails and ALPS releases (Emanuele)<br /> ** VisTrails latex package was extended so that python is no longer required. If python is not present in the system, instead of issuing a request to execute the workflow and retrieve the image from the Web, a local, previously saved image will be used. It is now also possible to embed the images of workflows into the latex file. Documentation about these and other features are currently in the README file and examples (example.tex) provided with the latex package.<br /> ** Windows and Mac beta binaries already include ALPS<br /> ** Are we distributing ALPS as part of VisTrails then? This seems backwards. ALPS should VisTrails as part of their stuff, no?<br /> *** '''We will have separate binary distributions: one with and one without ALPS'''<br /> <br /> * Is a multi-touch interface to VisTrails possible now that it's supported by Qt? (Phillip)<br /> ** Yes, Qt 4.6+ supports multi-tocuh --- '''Huy will look into this; the goal is to make this work both for the workflow builder/tree view and the spreadsheet'''<br /> ** We still need to figure out what makes sense for multi-touch, however, do we have requirements from ETH-Zurich?<br /> *** We want to map the zoom, pan, click, etc to multi-touch, so that VisTrails can be intractable on a multi-touch screen without a mouse<br /> <br /> * What is the status of the Vismashup i{Phone|Pad|Touch} app? (Phillip)<br /> ** If you mean mac binary, there's an alpha version here: http://www.sci.utah.edu/~emanuele/files/vismashup <br /> ** Do you mean iphone app? [Phillip: yes]<br /> ** '''Wendel will look into this'''<br /> *** One of the issues is how to effectively handle images that are larger than the memory on the iPods and iPads---we need to process these on the server or use a model that allows the image to be manipulated on the client<br /> *** We also need to connect the app with the crowdlabs server, so that the app can get the list of mashups<br /> *** Phillip will investigate the feasibility of working directly with JavaScript and bypass flash<br /> <br /> <br /> <br /> === Oct 19, 2010 ===<br /> <br /> * Update on Trac and Roadmap (Emanuele and David)<br /> ** Trac is now linked to git repository<br /> ** Roadmap on Trac has been cleaned up so we can hide completed milestones<br /> ** Tickets are being reassigned/revisited. Many tickets (24) are not associated with a milestone yet https://vistrails.sci.utah.edu/report/3<br /> * Synchronizing VisTrails and ALPS releases<br /> * Web services support (Tommy)<br /> ** discuss the interaction between the Web Services package and upgrades<br /> * Subworkflows update (Daniel)<br /> * Testing of new module drawing (Erik)<br /> * Aliases<br /> ** Currently, an alias is stored on a parameter. We need aliases to be stored at a higher level so that changing an alias is not a change to a parameter. The one issue is that there is a link between parmaeters and aliases in that an alias can only exist for versions that have the specified parameter. We might also have two versions where the alias points to different parameters. We could just store aliases as a root-level workflow element so that the set of aliases is versioned corrected.<br /> ** In the current implementation, the aliases parsed from the parameters are stored in a dictionary in the workflow, so storing aliases as a root-level workflows element might be the way to go --[[User:Emanuele|Emanuele]] 19:50, 15 October 2010 (MDT)<br /> * Error logging<br /> ** Suggest that we have a single point for all error messages to pass through that is linked to both core.debug and GUI elements that display error messages. This should improve our error handling significantly<br /> * PyQt dependencies in core<br /> ** Can we get rid of PyQt dependencies in core (and db)? This would probably require moving the signals/slots that we currently use to a similar Python implementation which shouldn't be too difficult. This would also require the specification of configuration widgets not as classes but rather some text that can be used to import the GUI elements only when using the GUI code. However, what would happen with workflows that run in command-line mode but require some graphical output?<br /> * Ports<br /> ** One annoying part of the current VisTrails model is that users are not given any visual indication that a port value is already set by a function or vice versa by a connection. In addition, we have no way to specify or enforce cardinality on ports at design time. It seems like we could allow developers to specify when a port should not be connected to more than one value (via a function or a connection), and give some visual feedback when a port has already been specified in one manner.<br /> <br /> ==== Meeting Notes ====<br /> <br /> * Bug regarding userpackages at initial vistrails startup causing startup to fail.<br /> ** Still an issue, ticket still open as a 1.6 milestone<br /> * Copy-paste bug<br /> ** Shortcut not working on Mac. Focus makes shortcut try to copy pipeline.<br /> ** Current fix appends to clipboard as users type - this is not a good fix.<br /> * Synchronizing ALPS and VisTrails releases<br /> ** 1.6 scheduled for Dec. 1<br /> * Webservices<br /> ** New library being used, but it's very low level.<br /> ** Simple types may cause problems with some web-services using complex or XML-based types.<br /> ** Need to make sure that this library can handle at least MOST of the web-services out there.<br /> * Sub-workflows<br /> ** Daniel's changes seem functional and very slick.<br /> ** A little more work on it seems necessary.<br /> ** Need some easy GUI mechanism to delete a subworkflow.<br /> ** Is the exclamation point in the module draw the best way to handle this? Make a tool-tip to explain it.<br /> * VTK Wrapping<br /> ** Changes to VTK Python wrapping is causing headaches - particularly in backwards compatibility.<br /> ** Method typing has changed to expect things like "List" or "Vector" - makes things hard on the user when defining these inputs.<br /> ** Removing List and Vector types and replacing them with dynamically generated versions from Tuples.<br /> * Aliases - Agenda pushed until after Vis.<br /> * Error Logging - Unify some error handling to improve how exceptions are dealt with. More on this after Vis.<br /> * Refactoring out PyQt dependencies from Core/Db/etc - Need to move all the signals/slots into GUI elements. Questions come when a workflow uses GUI elements as inputs. - More after vis.<br /> * Ports - We need a way of showing how a port is specified - function vs. input port. Cardinality must be established in these cases. Ordering on multiple connections should be handled in some way - whether it's just a documentation issue or a more fundamental one.<br /> ** Should we have a tutorial mode that bring up tooltips when someone does something new as if it were a live tutorial.<br /> <br /> <br /> === Oct 12, 2010 ===<br /> <br /> * Trac and Roadmap (Emanuele): Emanuele suggested we should create a development roadmap and make it a permanent item on the agenda. The idea is to go through the Trac tickets and use them to build the roadmap that would be made ''public''.<br /> ** go through trac and sort by priority<br /> ** add other suggestions to roadmap as well<br /> ** admins can add milestones via the admin tab<br /> ** ''David will assign tasks on list; those assigned need to follow up and check and fix them''<br /> <br /> * Update on [[PythonSource error reporting and logging]] (Tommy)<br /> ** ''Need to make sure that all messages are printed through debug.X, so that the debug level (verboseness in preferences) is correctly used''<br /> ** Can cut/copy/paste now<br /> ** Can we prevent the delete from happening?<br /> ** Phillip noted the spyder project has an interactive python console: http://packages.python.org/spyder/<br /> ** Can press enter to get to the end<br /> ** Need to check if this works or if we need to change<br /> ** Now have ability to see stack trace from the GUI (triangle menu)<br /> ** ''Suggest using a dialog to display trace when this item is selected instead of printing to console''<br /> ** Also saving stack trace to the execution log<br /> ** Also printing debug information to vistrails log<br /> ** ''Add trac item to fix prints to use debug''<br /> ** ''Try to fix core.debug to have gui.debug to eliminate Qt dependency''<br /> <br /> * Update on subworkflow (Daniel)<br /> ** Notification when subworkflow is outdated triggered<br /> ** new_abstraction signal from registry in addition to new_module signal so that we can check if we need subworkflows to update<br /> ** How to display the possible upgrades (can be upgraded, can be upgraded but may break, etc.)<br /> <br /> * New utility added by Carlos (currently at scripts/module_appearance) to draw fringes visually, which generates output that can be pasted directly into the add_module call. <br /> ** ''Erik will test this on Mac and Windows.''<br /> <br /> * Preparing a "Get Started" tutorial for SIGMOD repeatability <br /> ** Do we have instructions on how to use the latex package without crowdlabs?<br /> *** links to actual workflows in latex<br /> *** have README and example for latex in the source<br /> ** Metadata associated with VisTrails: should we have the ability to add vistrail-level metadata? e.g., who created the vistrail, its purpose, etc.<br /> *** have pointers to paper to workflow and vistrails<br /> *** add pointers to papers<br /> *** allow access to add/edit annotations at the vistrail level<br /> *** ''add GUI element to allow people to edit/add annotations''<br /> <br /> * Maintaining VTK package<br /> ** have four classes that don't wrap, have error even in python level<br /> ** works well with VisTrails otherwise<br /> ** haven't checked the new changes from the wrapping<br /> ** ''Wendel will check on the new version''<br /> <br /> * [[Web services package]] (Tommy)<br /> <br /> === Oct 5, 2010 ===<br /> <br /> * Welcome Tommy!<br /> <br /> * Update on subworkflow (Daniel)<br /> <br /> * Issues raised by Matthias:<br /> <br /> - need to automatically upgrade subworkflows. Currently it is necessary to manually upgrade a subworkflow whenever the version of one of the modules inside them changes.<br /> <br /> - explorations do not detect an MplFigureCell embedded in a subworkflow.</div>

Mates https://www.vistrails.org//index.php?title=Development&diff=3655 Development 2010-11-03T19:51:37Z

<p>Mates: /* Nov 3, 2010 */</p> <hr /> <div>== Roadmap ==<br /> <br /> <br /> == Weekly Meetings ==<br /> <br /> === Nov 3, 2010 ===<br /> <br /> * Upgrading subworkflows (Dan)<br /> ** add updated versions to subworkflow vistrail and push to registry<br /> ** fixed bug with version/descriptor redirects<br /> ** key remaining task is to replace the box representing the subworkflow in the top-level workflow to reflect the change to the upgraded version of the subworkflow<br /> ** many corner cases, but we should get the vanilla upgrade (working) to git<br /> ** need to ensure that each subworkflow file has its own uuid; if a subworkflow exists in two different vistrails, it should have a different uuid (namespace).<br /> ** otherwise, we can get crosstalk where one vt can update the subworkflow from another vt (we can propgate changes to other vts via merges, but this should be a user choice...)<br /> ** enhancements:<br /> *** allow users to merge subworkflows from a file into their own subworkflow (to incorporate outside changes)<br /> *** latest version is the most recent non-upgraded version; if that version has an upgrade, use the upgrade<br /> * Any update on fixing VTK Package? It seems that the ParaView package has the same issues.<br /> * Synchronizing VisTrails and ALPS releases (Emanuele)<br /> ** Windows and Mac beta binaries already include ALPS<br /> ** Are we distributing ALPS as part of VisTrails then? This seems backwards. ALPS should VisTrails as part of their stuff, no?<br /> * Is a multi-touch interface to VisTrails possible now that it's supported by Qt? (Phillip)<br /> ** Yes, Qt 4.6+ supports multi-tocuh<br /> ** We still need to figure out what makes sense for multi-touch, however, do we have requirements from ETH-Zurich?<br /> *** We want to map the zoom, pan, click, etc to multi-touch, so that VisTrails can be intractable on a multi-touch screen without a mouse<br /> * What is the status of the Vismashup i{Phone|Pad|Touch} app? (Phillip)<br /> ** If you mean mac binary, there's an alpha version here: http://www.sci.utah.edu/~emanuele/files/vismashup-mac-10.5-intel-0.3-alpha-rev2219.dmg <br /> ** Do you mean iphone app? [Phillip: yes]<br /> * Data Publishing with crowdLabs. (Phillip)<br /> ** Do we want data publishing capabilities in VisTrails and crowdLabs?<br /> ** If so...<br /> *** Currently limited data provenance for data. HTTPFile module lacks meta data, RepoSync module has poor usability.<br /> *** Persistent Data works for experimental data. Can we use it for publishing data?<br /> **** Publishing data or storing data on the server and pushing back and forth?<br /> **** What is the current scheme for identifying data on the server?<br /> **** Do we need to do better than the ids, versions, hashes that the peristence package provides?<br /> *** Do we want to utilize data publishing software like [http://thedata.org/home The Dataverse Network] or maybe [http://www.fedora-commons.org/ Fedora Commons]?<br /> <br /> === Oct 19, 2010 ===<br /> <br /> * Update on Trac and Roadmap (Emanuele and David)<br /> ** Trac is now linked to git repository<br /> ** Roadmap on Trac has been cleaned up so we can hide completed milestones<br /> ** Tickets are being reassigned/revisited. Many tickets (24) are not associated with a milestone yet https://vistrails.sci.utah.edu/report/3<br /> * Synchronizing VisTrails and ALPS releases<br /> * Web services support (Tommy)<br /> ** discuss the interaction between the Web Services package and upgrades<br /> * Subworkflows update (Daniel)<br /> * Testing of new module drawing (Erik)<br /> * Aliases<br /> ** Currently, an alias is stored on a parameter. We need aliases to be stored at a higher level so that changing an alias is not a change to a parameter. The one issue is that there is a link between parmaeters and aliases in that an alias can only exist for versions that have the specified parameter. We might also have two versions where the alias points to different parameters. We could just store aliases as a root-level workflow element so that the set of aliases is versioned corrected.<br /> ** In the current implementation, the aliases parsed from the parameters are stored in a dictionary in the workflow, so storing aliases as a root-level workflows element might be the way to go --[[User:Emanuele|Emanuele]] 19:50, 15 October 2010 (MDT)<br /> * Error logging<br /> ** Suggest that we have a single point for all error messages to pass through that is linked to both core.debug and GUI elements that display error messages. This should improve our error handling significantly<br /> * PyQt dependencies in core<br /> ** Can we get rid of PyQt dependencies in core (and db)? This would probably require moving the signals/slots that we currently use to a similar Python implementation which shouldn't be too difficult. This would also require the specification of configuration widgets not as classes but rather some text that can be used to import the GUI elements only when using the GUI code. However, what would happen with workflows that run in command-line mode but require some graphical output?<br /> * Ports<br /> ** One annoying part of the current VisTrails model is that users are not given any visual indication that a port value is already set by a function or vice versa by a connection. In addition, we have no way to specify or enforce cardinality on ports at design time. It seems like we could allow developers to specify when a port should not be connected to more than one value (via a function or a connection), and give some visual feedback when a port has already been specified in one manner.<br /> <br /> ==== Meeting Notes ====<br /> <br /> * Bug regarding userpackages at initial vistrails startup causing startup to fail.<br /> ** Still an issue, ticket still open as a 1.6 milestone<br /> * Copy-paste bug<br /> ** Shortcut not working on Mac. Focus makes shortcut try to copy pipeline.<br /> ** Current fix appends to clipboard as users type - this is not a good fix.<br /> * Synchronizing ALPS and VisTrails releases<br /> ** 1.6 scheduled for Dec. 1<br /> * Webservices<br /> ** New library being used, but it's very low level.<br /> ** Simple types may cause problems with some web-services using complex or XML-based types.<br /> ** Need to make sure that this library can handle at least MOST of the web-services out there.<br /> * Sub-workflows<br /> ** Daniel's changes seem functional and very slick.<br /> ** A little more work on it seems necessary.<br /> ** Need some easy GUI mechanism to delete a subworkflow.<br /> ** Is the exclamation point in the module draw the best way to handle this? Make a tool-tip to explain it.<br /> * VTK Wrapping<br /> ** Changes to VTK Python wrapping is causing headaches - particularly in backwards compatibility.<br /> ** Method typing has changed to expect things like "List" or "Vector" - makes things hard on the user when defining these inputs.<br /> ** Removing List and Vector types and replacing them with dynamically generated versions from Tuples.<br /> * Aliases - Agenda pushed until after Vis.<br /> * Error Logging - Unify some error handling to improve how exceptions are dealt with. More on this after Vis.<br /> * Refactoring out PyQt dependencies from Core/Db/etc - Need to move all the signals/slots into GUI elements. Questions come when a workflow uses GUI elements as inputs. - More after vis.<br /> * Ports - We need a way of showing how a port is specified - function vs. input port. Cardinality must be established in these cases. Ordering on multiple connections should be handled in some way - whether it's just a documentation issue or a more fundamental one.<br /> ** Should we have a tutorial mode that bring up tooltips when someone does something new as if it were a live tutorial.<br /> <br /> <br /> === Oct 12, 2010 ===<br /> <br /> * Trac and Roadmap (Emanuele): Emanuele suggested we should create a development roadmap and make it a permanent item on the agenda. The idea is to go through the Trac tickets and use them to build the roadmap that would be made ''public''.<br /> ** go through trac and sort by priority<br /> ** add other suggestions to roadmap as well<br /> ** admins can add milestones via the admin tab<br /> ** ''David will assign tasks on list; those assigned need to follow up and check and fix them''<br /> <br /> * Update on [[PythonSource error reporting and logging]] (Tommy)<br /> ** ''Need to make sure that all messages are printed through debug.X, so that the debug level (verboseness in preferences) is correctly used''<br /> ** Can cut/copy/paste now<br /> ** Can we prevent the delete from happening?<br /> ** Phillip noted the spyder project has an interactive python console: http://packages.python.org/spyder/<br /> ** Can press enter to get to the end<br /> ** Need to check if this works or if we need to change<br /> ** Now have ability to see stack trace from the GUI (triangle menu)<br /> ** ''Suggest using a dialog to display trace when this item is selected instead of printing to console''<br /> ** Also saving stack trace to the execution log<br /> ** Also printing debug information to vistrails log<br /> ** ''Add trac item to fix prints to use debug''<br /> ** ''Try to fix core.debug to have gui.debug to eliminate Qt dependency''<br /> <br /> * Update on subworkflow (Daniel)<br /> ** Notification when subworkflow is outdated triggered<br /> ** new_abstraction signal from registry in addition to new_module signal so that we can check if we need subworkflows to update<br /> ** How to display the possible upgrades (can be upgraded, can be upgraded but may break, etc.)<br /> <br /> * New utility added by Carlos (currently at scripts/module_appearance) to draw fringes visually, which generates output that can be pasted directly into the add_module call. <br /> ** ''Erik will test this on Mac and Windows.''<br /> <br /> * Preparing a "Get Started" tutorial for SIGMOD repeatability <br /> ** Do we have instructions on how to use the latex package without crowdlabs?<br /> *** links to actual workflows in latex<br /> *** have README and example for latex in the source<br /> ** Metadata associated with VisTrails: should we have the ability to add vistrail-level metadata? e.g., who created the vistrail, its purpose, etc.<br /> *** have pointers to paper to workflow and vistrails<br /> *** add pointers to papers<br /> *** allow access to add/edit annotations at the vistrail level<br /> *** ''add GUI element to allow people to edit/add annotations''<br /> <br /> * Maintaining VTK package<br /> ** have four classes that don't wrap, have error even in python level<br /> ** works well with VisTrails otherwise<br /> ** haven't checked the new changes from the wrapping<br /> ** ''Wendel will check on the new version''<br /> <br /> * [[Web services package]] (Tommy)<br /> <br /> === Oct 5, 2010 ===<br /> <br /> * Welcome Tommy!<br /> <br /> * Update on subworkflow (Daniel)<br /> <br /> * Issues raised by Matthias:<br /> <br /> - need to automatically upgrade subworkflows. Currently it is necessary to manually upgrade a subworkflow whenever the version of one of the modules inside them changes.<br /> <br /> - explorations do not detect an MplFigureCell embedded in a subworkflow.</div>

Mates https://www.vistrails.org//index.php?title=Development&diff=3652 Development 2010-11-03T19:05:14Z

<p>Mates: /* Nov 3, 2010 */</p> <hr /> <div>== Roadmap ==<br /> <br /> <br /> == Weekly Meetings ==<br /> <br /> === Nov 3, 2010 ===<br /> <br /> * Upgrading subworkflows (Dan)<br /> ** add updated versions to subworkflow vistrail and push to registry<br /> ** fixed bug with version/descriptor redirects<br /> ** key remaining task is to replace the box representing the subworkflow in the top-level workflow to reflect the change to the upgraded version of the subworkflow<br /> ** many corner cases, but we should get the vanilla upgrade (working) to git<br /> ** need to ensure that each subworkflow file has its own uuid; if a subworkflow exists in two different vistrails, it should have a different uuid (namespace).<br /> ** otherwise, we can get crosstalk where one vt can update the subworkflow from another vt (we can propgate changes to other vts via merges, but this should be a user choice...)<br /> ** enhancements:<br /> *** allow users to merge subworkflows from a file into their own subworkflow (to incorporate outside changes)<br /> *** latest version is the most recent non-upgraded version; if that version has an upgrade, use the upgrade<br /> * Any update on fixing VTK Package? It seems that the ParaView package has the same issues.<br /> * Synchronizing VisTrails and ALPS releases (Emanuele)<br /> ** Windows and Mac beta binaries already include ALPS<br /> * Is a multi-touch interface to VisTrails possible now that it's supported by Qt? (Phillip)<br /> * What is the status of the Vismashup i{Phone|Pad|Touch} app? (Phillip)<br /> ** If you mean mac binary, there's an alpha version here: http://www.sci.utah.edu/~emanuele/files/vismashup-mac-10.5-intel-0.3-alpha-rev2219.dmg <br /> ** Do you mean iphone app? [Phillip: yes]<br /> * Data Publishing with crowdLabs. (Phillip)<br /> ** Do we want data publishing capabilities in VisTrails and crowdLabs?<br /> ** If so...<br /> *** Currently limited data provenance for data. HTTPFile module lacks meta data, RepoSync module has poor usability.<br /> *** Persistent Data works for experimental data. Can we use it for publishing data?<br /> *** Do we want to utilize data publishing software like [http://thedata.org/home The Dataverse Network] or maybe [http://www.fedora-commons.org/ Fedora Commons]?<br /> <br /> === Oct 19, 2010 ===<br /> <br /> * Update on Trac and Roadmap (Emanuele and David)<br /> ** Trac is now linked to git repository<br /> ** Roadmap on Trac has been cleaned up so we can hide completed milestones<br /> ** Tickets are being reassigned/revisited. Many tickets (24) are not associated with a milestone yet https://vistrails.sci.utah.edu/report/3<br /> * Synchronizing VisTrails and ALPS releases<br /> * Web services support (Tommy)<br /> ** discuss the interaction between the Web Services package and upgrades<br /> * Subworkflows update (Daniel)<br /> * Testing of new module drawing (Erik)<br /> * Aliases<br /> ** Currently, an alias is stored on a parameter. We need aliases to be stored at a higher level so that changing an alias is not a change to a parameter. The one issue is that there is a link between parmaeters and aliases in that an alias can only exist for versions that have the specified parameter. We might also have two versions where the alias points to different parameters. We could just store aliases as a root-level workflow element so that the set of aliases is versioned corrected.<br /> ** In the current implementation, the aliases parsed from the parameters are stored in a dictionary in the workflow, so storing aliases as a root-level workflows element might be the way to go --[[User:Emanuele|Emanuele]] 19:50, 15 October 2010 (MDT)<br /> * Error logging<br /> ** Suggest that we have a single point for all error messages to pass through that is linked to both core.debug and GUI elements that display error messages. This should improve our error handling significantly<br /> * PyQt dependencies in core<br /> ** Can we get rid of PyQt dependencies in core (and db)? This would probably require moving the signals/slots that we currently use to a similar Python implementation which shouldn't be too difficult. This would also require the specification of configuration widgets not as classes but rather some text that can be used to import the GUI elements only when using the GUI code. However, what would happen with workflows that run in command-line mode but require some graphical output?<br /> * Ports<br /> ** One annoying part of the current VisTrails model is that users are not given any visual indication that a port value is already set by a function or vice versa by a connection. In addition, we have no way to specify or enforce cardinality on ports at design time. It seems like we could allow developers to specify when a port should not be connected to more than one value (via a function or a connection), and give some visual feedback when a port has already been specified in one manner.<br /> <br /> ==== Meeting Notes ====<br /> <br /> * Bug regarding userpackages at initial vistrails startup causing startup to fail.<br /> ** Still an issue, ticket still open as a 1.6 milestone<br /> * Copy-paste bug<br /> ** Shortcut not working on Mac. Focus makes shortcut try to copy pipeline.<br /> ** Current fix appends to clipboard as users type - this is not a good fix.<br /> * Synchronizing ALPS and VisTrails releases<br /> ** 1.6 scheduled for Dec. 1<br /> * Webservices<br /> ** New library being used, but it's very low level.<br /> ** Simple types may cause problems with some web-services using complex or XML-based types.<br /> ** Need to make sure that this library can handle at least MOST of the web-services out there.<br /> * Sub-workflows<br /> ** Daniel's changes seem functional and very slick.<br /> ** A little more work on it seems necessary.<br /> ** Need some easy GUI mechanism to delete a subworkflow.<br /> ** Is the exclamation point in the module draw the best way to handle this? Make a tool-tip to explain it.<br /> * VTK Wrapping<br /> ** Changes to VTK Python wrapping is causing headaches - particularly in backwards compatibility.<br /> ** Method typing has changed to expect things like "List" or "Vector" - makes things hard on the user when defining these inputs.<br /> ** Removing List and Vector types and replacing them with dynamically generated versions from Tuples.<br /> * Aliases - Agenda pushed until after Vis.<br /> * Error Logging - Unify some error handling to improve how exceptions are dealt with. More on this after Vis.<br /> * Refactoring out PyQt dependencies from Core/Db/etc - Need to move all the signals/slots into GUI elements. Questions come when a workflow uses GUI elements as inputs. - More after vis.<br /> * Ports - We need a way of showing how a port is specified - function vs. input port. Cardinality must be established in these cases. Ordering on multiple connections should be handled in some way - whether it's just a documentation issue or a more fundamental one.<br /> ** Should we have a tutorial mode that bring up tooltips when someone does something new as if it were a live tutorial.<br /> <br /> <br /> === Oct 12, 2010 ===<br /> <br /> * Trac and Roadmap (Emanuele): Emanuele suggested we should create a development roadmap and make it a permanent item on the agenda. The idea is to go through the Trac tickets and use them to build the roadmap that would be made ''public''.<br /> ** go through trac and sort by priority<br /> ** add other suggestions to roadmap as well<br /> ** admins can add milestones via the admin tab<br /> ** ''David will assign tasks on list; those assigned need to follow up and check and fix them''<br /> <br /> * Update on [[PythonSource error reporting and logging]] (Tommy)<br /> ** ''Need to make sure that all messages are printed through debug.X, so that the debug level (verboseness in preferences) is correctly used''<br /> ** Can cut/copy/paste now<br /> ** Can we prevent the delete from happening?<br /> ** Phillip noted the spyder project has an interactive python console: http://packages.python.org/spyder/<br /> ** Can press enter to get to the end<br /> ** Need to check if this works or if we need to change<br /> ** Now have ability to see stack trace from the GUI (triangle menu)<br /> ** ''Suggest using a dialog to display trace when this item is selected instead of printing to console''<br /> ** Also saving stack trace to the execution log<br /> ** Also printing debug information to vistrails log<br /> ** ''Add trac item to fix prints to use debug''<br /> ** ''Try to fix core.debug to have gui.debug to eliminate Qt dependency''<br /> <br /> * Update on subworkflow (Daniel)<br /> ** Notification when subworkflow is outdated triggered<br /> ** new_abstraction signal from registry in addition to new_module signal so that we can check if we need subworkflows to update<br /> ** How to display the possible upgrades (can be upgraded, can be upgraded but may break, etc.)<br /> <br /> * New utility added by Carlos (currently at scripts/module_appearance) to draw fringes visually, which generates output that can be pasted directly into the add_module call. <br /> ** ''Erik will test this on Mac and Windows.''<br /> <br /> * Preparing a "Get Started" tutorial for SIGMOD repeatability <br /> ** Do we have instructions on how to use the latex package without crowdlabs?<br /> *** links to actual workflows in latex<br /> *** have README and example for latex in the source<br /> ** Metadata associated with VisTrails: should we have the ability to add vistrail-level metadata? e.g., who created the vistrail, its purpose, etc.<br /> *** have pointers to paper to workflow and vistrails<br /> *** add pointers to papers<br /> *** allow access to add/edit annotations at the vistrail level<br /> *** ''add GUI element to allow people to edit/add annotations''<br /> <br /> * Maintaining VTK package<br /> ** have four classes that don't wrap, have error even in python level<br /> ** works well with VisTrails otherwise<br /> ** haven't checked the new changes from the wrapping<br /> ** ''Wendel will check on the new version''<br /> <br /> * [[Web services package]] (Tommy)<br /> <br /> === Oct 5, 2010 ===<br /> <br /> * Welcome Tommy!<br /> <br /> * Update on subworkflow (Daniel)<br /> <br /> * Issues raised by Matthias:<br /> <br /> - need to automatically upgrade subworkflows. Currently it is necessary to manually upgrade a subworkflow whenever the version of one of the modules inside them changes.<br /> <br /> - explorations do not detect an MplFigureCell embedded in a subworkflow.</div>

Mates https://www.vistrails.org//index.php?title=Development&diff=3649 Development 2010-11-03T18:05:44Z

<p>Mates: /* Nov 3, 2010 */</p> <hr /> <div>== Roadmap ==<br /> <br /> <br /> == Weekly Meetings ==<br /> <br /> === Nov 3, 2010 ===<br /> <br /> * Upgrading subworkflows (Dan)<br /> ** add updated versions to subworkflow vistrail and push to registry<br /> ** fixed bug with version/descriptor redirects<br /> ** key remaining task is to replace the box representing the subworkflow in the top-level workflow to reflect the change to the upgraded version of the subworkflow<br /> ** many corner cases, but we should get the vanilla upgrade (working) to git<br /> ** need to ensure that each subworkflow file has its own uuid; if a subworkflow exists in two different vistrails, it should have a different uuid (namespace).<br /> ** otherwise, we can get crosstalk where one vt can update the subworkflow from another vt (we can propgate changes to other vts via merges, but this should be a user choice...)<br /> ** enhancements:<br /> *** allow users to merge subworkflows from a file into their own subworkflow (to incorporate outside changes)<br /> *** latest version is the most recent non-upgraded version; if that version has an upgrade, use the upgrade<br /> * Any update on fixing VTK Package? It seems that the ParaView package has the same issues.<br /> * Synchronizing VisTrails and ALPS releases (Emanuele)<br /> ** Windows and Mac beta binaries already include ALPS<br /> * Is a multi-touch interface to VisTrails possible now that it's supported by Qt? (Phillip)<br /> * What is the status of the Vismashup apple app? (Phillip)<br /> * Data Publishing with crowdLabs. (Phillip)<br /> ** Do we want data publishing capabilities in VisTrails and crowdLabs?<br /> ** If so...<br /> *** Currently limited data provenance for data. HTTPFile module lacks meta data, RepoSync module has poor usability.<br /> *** Persistent Data works for experimental data. Can we use it for publishing data?<br /> *** Do we want to utilize data publishing software like [http://thedata.org/home The Dataverse Network] or maybe [http://www.fedora-commons.org/ Fedora Commons]?<br /> <br /> === Oct 19, 2010 ===<br /> <br /> * Update on Trac and Roadmap (Emanuele and David)<br /> ** Trac is now linked to git repository<br /> ** Roadmap on Trac has been cleaned up so we can hide completed milestones<br /> ** Tickets are being reassigned/revisited. Many tickets (24) are not associated with a milestone yet https://vistrails.sci.utah.edu/report/3<br /> * Synchronizing VisTrails and ALPS releases<br /> * Web services support (Tommy)<br /> ** discuss the interaction between the Web Services package and upgrades<br /> * Subworkflows update (Daniel)<br /> * Testing of new module drawing (Erik)<br /> * Aliases<br /> ** Currently, an alias is stored on a parameter. We need aliases to be stored at a higher level so that changing an alias is not a change to a parameter. The one issue is that there is a link between parmaeters and aliases in that an alias can only exist for versions that have the specified parameter. We might also have two versions where the alias points to different parameters. We could just store aliases as a root-level workflow element so that the set of aliases is versioned corrected.<br /> ** In the current implementation, the aliases parsed from the parameters are stored in a dictionary in the workflow, so storing aliases as a root-level workflows element might be the way to go --[[User:Emanuele|Emanuele]] 19:50, 15 October 2010 (MDT)<br /> * Error logging<br /> ** Suggest that we have a single point for all error messages to pass through that is linked to both core.debug and GUI elements that display error messages. This should improve our error handling significantly<br /> * PyQt dependencies in core<br /> ** Can we get rid of PyQt dependencies in core (and db)? This would probably require moving the signals/slots that we currently use to a similar Python implementation which shouldn't be too difficult. This would also require the specification of configuration widgets not as classes but rather some text that can be used to import the GUI elements only when using the GUI code. However, what would happen with workflows that run in command-line mode but require some graphical output?<br /> * Ports<br /> ** One annoying part of the current VisTrails model is that users are not given any visual indication that a port value is already set by a function or vice versa by a connection. In addition, we have no way to specify or enforce cardinality on ports at design time. It seems like we could allow developers to specify when a port should not be connected to more than one value (via a function or a connection), and give some visual feedback when a port has already been specified in one manner.<br /> <br /> ==== Meeting Notes ====<br /> <br /> * Bug regarding userpackages at initial vistrails startup causing startup to fail.<br /> ** Still an issue, ticket still open as a 1.6 milestone<br /> * Copy-paste bug<br /> ** Shortcut not working on Mac. Focus makes shortcut try to copy pipeline.<br /> ** Current fix appends to clipboard as users type - this is not a good fix.<br /> * Synchronizing ALPS and VisTrails releases<br /> ** 1.6 scheduled for Dec. 1<br /> * Webservices<br /> ** New library being used, but it's very low level.<br /> ** Simple types may cause problems with some web-services using complex or XML-based types.<br /> ** Need to make sure that this library can handle at least MOST of the web-services out there.<br /> * Sub-workflows<br /> ** Daniel's changes seem functional and very slick.<br /> ** A little more work on it seems necessary.<br /> ** Need some easy GUI mechanism to delete a subworkflow.<br /> ** Is the exclamation point in the module draw the best way to handle this? Make a tool-tip to explain it.<br /> * VTK Wrapping<br /> ** Changes to VTK Python wrapping is causing headaches - particularly in backwards compatibility.<br /> ** Method typing has changed to expect things like "List" or "Vector" - makes things hard on the user when defining these inputs.<br /> ** Removing List and Vector types and replacing them with dynamically generated versions from Tuples.<br /> * Aliases - Agenda pushed until after Vis.<br /> * Error Logging - Unify some error handling to improve how exceptions are dealt with. More on this after Vis.<br /> * Refactoring out PyQt dependencies from Core/Db/etc - Need to move all the signals/slots into GUI elements. Questions come when a workflow uses GUI elements as inputs. - More after vis.<br /> * Ports - We need a way of showing how a port is specified - function vs. input port. Cardinality must be established in these cases. Ordering on multiple connections should be handled in some way - whether it's just a documentation issue or a more fundamental one.<br /> ** Should we have a tutorial mode that bring up tooltips when someone does something new as if it were a live tutorial.<br /> <br /> <br /> === Oct 12, 2010 ===<br /> <br /> * Trac and Roadmap (Emanuele): Emanuele suggested we should create a development roadmap and make it a permanent item on the agenda. The idea is to go through the Trac tickets and use them to build the roadmap that would be made ''public''.<br /> ** go through trac and sort by priority<br /> ** add other suggestions to roadmap as well<br /> ** admins can add milestones via the admin tab<br /> ** ''David will assign tasks on list; those assigned need to follow up and check and fix them''<br /> <br /> * Update on [[PythonSource error reporting and logging]] (Tommy)<br /> ** ''Need to make sure that all messages are printed through debug.X, so that the debug level (verboseness in preferences) is correctly used''<br /> ** Can cut/copy/paste now<br /> ** Can we prevent the delete from happening?<br /> ** Phillip noted the spyder project has an interactive python console: http://packages.python.org/spyder/<br /> ** Can press enter to get to the end<br /> ** Need to check if this works or if we need to change<br /> ** Now have ability to see stack trace from the GUI (triangle menu)<br /> ** ''Suggest using a dialog to display trace when this item is selected instead of printing to console''<br /> ** Also saving stack trace to the execution log<br /> ** Also printing debug information to vistrails log<br /> ** ''Add trac item to fix prints to use debug''<br /> ** ''Try to fix core.debug to have gui.debug to eliminate Qt dependency''<br /> <br /> * Update on subworkflow (Daniel)<br /> ** Notification when subworkflow is outdated triggered<br /> ** new_abstraction signal from registry in addition to new_module signal so that we can check if we need subworkflows to update<br /> ** How to display the possible upgrades (can be upgraded, can be upgraded but may break, etc.)<br /> <br /> * New utility added by Carlos (currently at scripts/module_appearance) to draw fringes visually, which generates output that can be pasted directly into the add_module call. <br /> ** ''Erik will test this on Mac and Windows.''<br /> <br /> * Preparing a "Get Started" tutorial for SIGMOD repeatability <br /> ** Do we have instructions on how to use the latex package without crowdlabs?<br /> *** links to actual workflows in latex<br /> *** have README and example for latex in the source<br /> ** Metadata associated with VisTrails: should we have the ability to add vistrail-level metadata? e.g., who created the vistrail, its purpose, etc.<br /> *** have pointers to paper to workflow and vistrails<br /> *** add pointers to papers<br /> *** allow access to add/edit annotations at the vistrail level<br /> *** ''add GUI element to allow people to edit/add annotations''<br /> <br /> * Maintaining VTK package<br /> ** have four classes that don't wrap, have error even in python level<br /> ** works well with VisTrails otherwise<br /> ** haven't checked the new changes from the wrapping<br /> ** ''Wendel will check on the new version''<br /> <br /> * [[Web services package]] (Tommy)<br /> <br /> === Oct 5, 2010 ===<br /> <br /> * Welcome Tommy!<br /> <br /> * Update on subworkflow (Daniel)<br /> <br /> * Issues raised by Matthias:<br /> <br /> - need to automatically upgrade subworkflows. Currently it is necessary to manually upgrade a subworkflow whenever the version of one of the modules inside them changes.<br /> <br /> - explorations do not detect an MplFigureCell embedded in a subworkflow.</div>

Mates https://www.vistrails.org//index.php?title=Development&diff=3648 Development 2010-11-03T17:50:17Z

<p>Mates: /* Nov 3, 2010 */</p> <hr /> <div>== Roadmap ==<br /> <br /> <br /> == Weekly Meetings ==<br /> <br /> === Nov 3, 2010 ===<br /> <br /> * Upgrading subworkflows (Dan)<br /> ** add updated versions to subworkflow vistrail and push to registry<br /> ** fixed bug with version/descriptor redirects<br /> ** key remaining task is to replace the box representing the subworkflow in the top-level workflow to reflect the change to the upgraded version of the subworkflow<br /> ** many corner cases, but we should get the vanilla upgrade (working) to git<br /> ** need to ensure that each subworkflow file has its own uuid; if a subworkflow exists in two different vistrails, it should have a different uuid (namespace).<br /> ** otherwise, we can get crosstalk where one vt can update the subworkflow from another vt (we can propgate changes to other vts via merges, but this should be a user choice...)<br /> ** enhancements:<br /> *** allow users to merge subworkflows from a file into their own subworkflow (to incorporate outside changes)<br /> *** latest version is the most recent non-upgraded version; if that version has an upgrade, use the upgrade<br /> * Any update on fixing VTK Package? It seems that the ParaView package has the same issues.<br /> * Synchronizing VisTrails and ALPS releases (Emanuele)<br /> ** Windows and Mac beta binaries already include ALPS<br /> * Is a multi-touch interface to VisTrails possible now that it's supported by Qt? (Phillip)<br /> * What is the status of the Vismashup apple app? (Phillip)<br /> <br /> === Oct 19, 2010 ===<br /> <br /> * Update on Trac and Roadmap (Emanuele and David)<br /> ** Trac is now linked to git repository<br /> ** Roadmap on Trac has been cleaned up so we can hide completed milestones<br /> ** Tickets are being reassigned/revisited. Many tickets (24) are not associated with a milestone yet https://vistrails.sci.utah.edu/report/3<br /> * Synchronizing VisTrails and ALPS releases<br /> * Web services support (Tommy)<br /> ** discuss the interaction between the Web Services package and upgrades<br /> * Subworkflows update (Daniel)<br /> * Testing of new module drawing (Erik)<br /> * Aliases<br /> ** Currently, an alias is stored on a parameter. We need aliases to be stored at a higher level so that changing an alias is not a change to a parameter. The one issue is that there is a link between parmaeters and aliases in that an alias can only exist for versions that have the specified parameter. We might also have two versions where the alias points to different parameters. We could just store aliases as a root-level workflow element so that the set of aliases is versioned corrected.<br /> ** In the current implementation, the aliases parsed from the parameters are stored in a dictionary in the workflow, so storing aliases as a root-level workflows element might be the way to go --[[User:Emanuele|Emanuele]] 19:50, 15 October 2010 (MDT)<br /> * Error logging<br /> ** Suggest that we have a single point for all error messages to pass through that is linked to both core.debug and GUI elements that display error messages. This should improve our error handling significantly<br /> * PyQt dependencies in core<br /> ** Can we get rid of PyQt dependencies in core (and db)? This would probably require moving the signals/slots that we currently use to a similar Python implementation which shouldn't be too difficult. This would also require the specification of configuration widgets not as classes but rather some text that can be used to import the GUI elements only when using the GUI code. However, what would happen with workflows that run in command-line mode but require some graphical output?<br /> * Ports<br /> ** One annoying part of the current VisTrails model is that users are not given any visual indication that a port value is already set by a function or vice versa by a connection. In addition, we have no way to specify or enforce cardinality on ports at design time. It seems like we could allow developers to specify when a port should not be connected to more than one value (via a function or a connection), and give some visual feedback when a port has already been specified in one manner.<br /> <br /> ==== Meeting Notes ====<br /> <br /> * Bug regarding userpackages at initial vistrails startup causing startup to fail.<br /> ** Still an issue, ticket still open as a 1.6 milestone<br /> * Copy-paste bug<br /> ** Shortcut not working on Mac. Focus makes shortcut try to copy pipeline.<br /> ** Current fix appends to clipboard as users type - this is not a good fix.<br /> * Synchronizing ALPS and VisTrails releases<br /> ** 1.6 scheduled for Dec. 1<br /> * Webservices<br /> ** New library being used, but it's very low level.<br /> ** Simple types may cause problems with some web-services using complex or XML-based types.<br /> ** Need to make sure that this library can handle at least MOST of the web-services out there.<br /> * Sub-workflows<br /> ** Daniel's changes seem functional and very slick.<br /> ** A little more work on it seems necessary.<br /> ** Need some easy GUI mechanism to delete a subworkflow.<br /> ** Is the exclamation point in the module draw the best way to handle this? Make a tool-tip to explain it.<br /> * VTK Wrapping<br /> ** Changes to VTK Python wrapping is causing headaches - particularly in backwards compatibility.<br /> ** Method typing has changed to expect things like "List" or "Vector" - makes things hard on the user when defining these inputs.<br /> ** Removing List and Vector types and replacing them with dynamically generated versions from Tuples.<br /> * Aliases - Agenda pushed until after Vis.<br /> * Error Logging - Unify some error handling to improve how exceptions are dealt with. More on this after Vis.<br /> * Refactoring out PyQt dependencies from Core/Db/etc - Need to move all the signals/slots into GUI elements. Questions come when a workflow uses GUI elements as inputs. - More after vis.<br /> * Ports - We need a way of showing how a port is specified - function vs. input port. Cardinality must be established in these cases. Ordering on multiple connections should be handled in some way - whether it's just a documentation issue or a more fundamental one.<br /> ** Should we have a tutorial mode that bring up tooltips when someone does something new as if it were a live tutorial.<br /> <br /> <br /> === Oct 12, 2010 ===<br /> <br /> * Trac and Roadmap (Emanuele): Emanuele suggested we should create a development roadmap and make it a permanent item on the agenda. The idea is to go through the Trac tickets and use them to build the roadmap that would be made ''public''.<br /> ** go through trac and sort by priority<br /> ** add other suggestions to roadmap as well<br /> ** admins can add milestones via the admin tab<br /> ** ''David will assign tasks on list; those assigned need to follow up and check and fix them''<br /> <br /> * Update on [[PythonSource error reporting and logging]] (Tommy)<br /> ** ''Need to make sure that all messages are printed through debug.X, so that the debug level (verboseness in preferences) is correctly used''<br /> ** Can cut/copy/paste now<br /> ** Can we prevent the delete from happening?<br /> ** Phillip noted the spyder project has an interactive python console: http://packages.python.org/spyder/<br /> ** Can press enter to get to the end<br /> ** Need to check if this works or if we need to change<br /> ** Now have ability to see stack trace from the GUI (triangle menu)<br /> ** ''Suggest using a dialog to display trace when this item is selected instead of printing to console''<br /> ** Also saving stack trace to the execution log<br /> ** Also printing debug information to vistrails log<br /> ** ''Add trac item to fix prints to use debug''<br /> ** ''Try to fix core.debug to have gui.debug to eliminate Qt dependency''<br /> <br /> * Update on subworkflow (Daniel)<br /> ** Notification when subworkflow is outdated triggered<br /> ** new_abstraction signal from registry in addition to new_module signal so that we can check if we need subworkflows to update<br /> ** How to display the possible upgrades (can be upgraded, can be upgraded but may break, etc.)<br /> <br /> * New utility added by Carlos (currently at scripts/module_appearance) to draw fringes visually, which generates output that can be pasted directly into the add_module call. <br /> ** ''Erik will test this on Mac and Windows.''<br /> <br /> * Preparing a "Get Started" tutorial for SIGMOD repeatability <br /> ** Do we have instructions on how to use the latex package without crowdlabs?<br /> *** links to actual workflows in latex<br /> *** have README and example for latex in the source<br /> ** Metadata associated with VisTrails: should we have the ability to add vistrail-level metadata? e.g., who created the vistrail, its purpose, etc.<br /> *** have pointers to paper to workflow and vistrails<br /> *** add pointers to papers<br /> *** allow access to add/edit annotations at the vistrail level<br /> *** ''add GUI element to allow people to edit/add annotations''<br /> <br /> * Maintaining VTK package<br /> ** have four classes that don't wrap, have error even in python level<br /> ** works well with VisTrails otherwise<br /> ** haven't checked the new changes from the wrapping<br /> ** ''Wendel will check on the new version''<br /> <br /> * [[Web services package]] (Tommy)<br /> <br /> === Oct 5, 2010 ===<br /> <br /> * Welcome Tommy!<br /> <br /> * Update on subworkflow (Daniel)<br /> <br /> * Issues raised by Matthias:<br /> <br /> - need to automatically upgrade subworkflows. Currently it is necessary to manually upgrade a subworkflow whenever the version of one of the modules inside them changes.<br /> <br /> - explorations do not detect an MplFigureCell embedded in a subworkflow.</div>

Mates https://www.vistrails.org//index.php?title=Cmop_rdfs2&diff=2113 Cmop rdfs2 2009-09-15T15:33:20Z

<p>Mates: /* Presentation and Access */</p> <hr /> <div>== Task List ==<br /> <br /> In order to streamline the task management associated with re-creating the CMOP RDFS system, the sections below will be updated to provide instant glimpses of the current task lists for each of the various decoupled components.<br /> <br /> === Simulation and Forcings === <br /> <br /> Brad Grimm<br /> <br /> === Visualization and Analysis ===<br /> <br /> * Grid Construction from hgrid and vgrid<br /> ** Selfe from netCDF completed - working on other output types now<br /> <br /> * 2D plots from surface, bottom, and various depths<br /> ** Incorporating surface wind, etc<br /> <br /> * Transect Plots<br /> <br /> * Station Plots<br /> <br /> === Presentation and Access ===<br /> <br /> Basic Website Idea:<br><br /> * Establish an infrastructure that will enable the admin to create sections, subsections, and pages that point to vistrail workflows.<br><br /> * Sections will enable admin to organize the hierarchy of the navigation.<br><br /> * When a page is linked to a vistrail workflow, the web server grabs the output of that workflow. <br /> ** If the workflow is linked to a medley, the website will link the output images displayed to the medley.</div>

Mates https://www.vistrails.org//index.php?title=Cmop_rdfs2&diff=2112 Cmop rdfs2 2009-09-15T15:32:34Z

<p>Mates: /* Presentation and Access */</p> <hr /> <div>== Task List ==<br /> <br /> In order to streamline the task management associated with re-creating the CMOP RDFS system, the sections below will be updated to provide instant glimpses of the current task lists for each of the various decoupled components.<br /> <br /> === Simulation and Forcings === <br /> <br /> Brad Grimm<br /> <br /> === Visualization and Analysis ===<br /> <br /> * Grid Construction from hgrid and vgrid<br /> ** Selfe from netCDF completed - working on other output types now<br /> <br /> * 2D plots from surface, bottom, and various depths<br /> ** Incorporating surface wind, etc<br /> <br /> * Transect Plots<br /> <br /> * Station Plots<br /> <br /> === Presentation and Access ===<br /> <br /> Basic Website Idea:<br><br /> Establish an infrastructure that will enable the admin to create sections, subsections, and pages that point to vistrail workflows.<br><br /> Sections will enable admin to organize the hierarchy of the navigation.<br><br /> When a page is linked to a vistrail workflow, the web server grabs the output of that workflow. If the workflow is linked to a medley, the website will link the output images displayed to the medley.</div>

Mates https://www.vistrails.org//index.php?title=Cmop_rdfs2&diff=2103 Cmop rdfs2 2009-09-11T23:19:45Z

<p>Mates: /* Presentation and Access */</p> <hr /> <div>== Task List ==<br /> <br /> In order to streamline the task management associated with re-creating the CMOP RDFS system, the sections below will be updated to provide instant glimpses of the current task lists for each of the various decoupled components.<br /> <br /> === Simulation and Forcings === <br /> <br /> Brad Grimm<br /> <br /> === Visualization and Analysis ===<br /> <br /> * Grid Construction from hgrid and vgrid<br /> ** Selfe from netCDF completed - working on other output types now<br /> <br /> * 2D plots from surface, bottom, and various depths<br /> ** Incorporating surface wind, etc<br /> <br /> * Transect Plots<br /> <br /> * Station Plots<br /> <br /> === Presentation and Access ===<br /> <br /> Create basic website<br /> <br /> Models include:<br /> *Sections<br /> *Pages<br /> *Object (vistrail, workflow, medley)<br /> <br /> Create basic page views<br><br /> Connect to VisTrails server to retrieve visualization information</div>

Mates