SubworkflowsDesign - Revision history

Remi: /* Comments */

2013-09-11T18:11:36Z

Comments

← Older revision		Revision as of 18:11, 11 September 2013
Line 31:		Line 31:
	=== Comments ===		=== Comments ===

	[[#Introduction]] "A user must expand the group, make changes, possibly add back some Input/OutputPort modules, and then group the modules again."		[[#Introduction]] "A user must expand the group, make changes, possibly add back some Input/OutputPort modules, and then group the modules again."<br/>
			I didn't know you could create a group this way, by manually placing the InputPorts before grouping. However, ungrouping deletes them, so it does not look like a supported workflow. -- [RR]

	~~I didn't know you~~ could ~~create a group this way, by manually placing~~ the ~~InputPorts before grouping~~. ~~However, ungrouping deletes them, so~~ it ~~does not look like a supported workflow~~.		:Can we make the ungrouping preserve the input/output ports? Maybe even keep the connections to other modules? This would make it trivial to ungroup-edit-regroup. Editing in a new window would be even better and could work the same. But it would require more work. [TE]

	~~Can~~ we ~~make the ungrouping preserve the input/output ports~~? ~~Maybe even keep the connections to other modules~~? ~~This would make~~ it ~~trivial~~ to ~~ungroup-edit-regroup~~. ~~Editing in a new window would be even better~~ and ~~could work~~ the ~~same. But it would require more work.~~ [TE]		Are abstractions so broken we need to change them in 2.1? Is there a known example that breaks? Would it be enough to warn of the limitations in 2.1 and aim for the uuid version? [TE]

	~~Are abstractions so broken we need~~ to ~~change them~~ in 2.~~1? Is there a known example that breaks? Would~~ it ~~be enough to warn of the limitations in 2~~.~~1 and aim for the uuid version~~? ~~[TE~~]		:I added to trac some of the issues I encountered while building the "primes" example with subworkflows (it is currently checked-in with Groups). I think they are it: [https://www.vistrails.org/query?status=!closed&keywords=~abstraction trac: !closed abstraction]

Tommy: /* Comments */

2013-09-10T12:31:52Z

Comments

← Older revision		Revision as of 12:31, 10 September 2013
Line 36:		Line 36:

	Can we make the ungrouping preserve the input/output ports? Maybe even keep the connections to other modules? This would make it trivial to ungroup-edit-regroup. Editing in a new window would be even better and could work the same. But it would require more work. [TE]		Can we make the ungrouping preserve the input/output ports? Maybe even keep the connections to other modules? This would make it trivial to ungroup-edit-regroup. Editing in a new window would be even better and could work the same. But it would require more work. [TE]

			Are abstractions so broken we need to change them in 2.1? Is there a known example that breaks? Would it be enough to warn of the limitations in 2.1 and aim for the uuid version? [TE]

Tommy: /* Comments */

2013-09-10T12:28:44Z

Comments

← Older revision		Revision as of 12:28, 10 September 2013
Line 34:		Line 34:

	I didn't know you could create a group this way, by manually placing the InputPorts before grouping. However, ungrouping deletes them, so it does not look like a supported workflow.		I didn't know you could create a group this way, by manually placing the InputPorts before grouping. However, ungrouping deletes them, so it does not look like a supported workflow.

			Can we make the ungrouping preserve the input/output ports? Maybe even keep the connections to other modules? This would make it trivial to ungroup-edit-regroup. Editing in a new window would be even better and could work the same. But it would require more work. [TE]

Remi: /* Comments */

2013-09-06T14:20:18Z

Comments

← Older revision		Revision as of 14:20, 6 September 2013
Line 26:		Line 26:

	Subworkflows already use some uuids, but with a uuid for version ids (as in the uuid branch), the issues with identifying which version of a subworkflow (even across multiple files) are largely removed since the uuid will be unique. There still may be some interesting issues about which is the "most recent" version is since the ids are not monotonic.		Subworkflows already use some uuids, but with a uuid for version ids (as in the uuid branch), the issues with identifying which version of a subworkflow (even across multiple files) are largely removed since the uuid will be unique. There still may be some interesting issues about which is the "most recent" version is since the ids are not monotonic.



			=== Comments ===

			[[#Introduction]] "A user must expand the group, make changes, possibly add back some Input/OutputPort modules, and then group the modules again."

			I didn't know you could create a group this way, by manually placing the InputPorts before grouping. However, ungrouping deletes them, so it does not look like a supported workflow.

Remi at 14:12, 6 September 2013

2013-09-06T14:12:19Z

← Older revision		Revision as of 14:12, 6 September 2013
Line 1:		Line 1:
	=== Introduction ===		=== Introduction ===

	VisTrails has groups which are "encapsulate by value" entities and subworkflows which are "encapsulate by reference". A group is stored as a workflow with modules and connections (including special InputPort and OutputPort modules that determine which ports are exposed in the group module) while a subworkflow is stored as a version of a workflow in a vistrail. Thus, a change to a group will generate an entirely new workflow that is fully specified while a change to a subworkflow uses change-based provenance to only store the actions between the previous workflow and the new one. Groups are fairly simple and work well to abstract a piece of a workflow but use storage inefficiently. In addition, in their current implementation in VisTrails, they are impossible to edit in place. A user must expand the group, make changes, possibly add back some Input/OutputPort modules, and then group the modules again. Subworkflows solve the storage issue but introduce significant complexity because they must be indexed by the module registry. Subworkflows can exist locally (in a user's .vistrails directory), as part of a distributed package (can use _subworkflows list or add_subworkflow), or as part of a vt bundle. In addition, any reference to a subworkflow in a workflow must both identify the subworkflow and the version to be used.		VisTrails has groups which are "encapsulate by value" entities and subworkflows which are "encapsulate by reference". A group is stored as a workflow with modules and connections (including special InputPort and OutputPort modules that determine which ports are exposed in the group module) while a subworkflow is stored as a version of a workflow in a different vistrail. Thus, a change to a group will generate an entirely new workflow that is fully specified while a change to a subworkflow uses change-based provenance to only store the actions between the previous workflow and the new one. Groups are fairly simple and work well to abstract a piece of a workflow but use storage inefficiently. In addition, in their current implementation in VisTrails, they are impossible to edit in place. A user must expand the group, make changes, possibly add back some Input/OutputPort modules, and then group the modules again. Subworkflows solve the storage issue but introduce significant complexity because they must be indexed by the module registry. Subworkflows can exist locally (in a user's .vistrails directory), as part of a distributed package (can use _subworkflows list or add_subworkflow), or as part of a vt bundle. In addition, any reference to a subworkflow in a workflow must both identify the subworkflow and the version to be used.

	=== Group Edits ===		=== Group Edits ===
Line 12:		Line 12:
	=== Subworkflow versions and upgrades ===		=== Subworkflow versions and upgrades ===

	Because we want to allow any user to run a workflow that contains a subworkflow, we save ~~any~~ subworkflows in the vt bundle. This ensures that a user who receives this bundle can run the workflow, assuming ~~their~~ have all the necessary packages installed. At the same time, this means that there can be many different versions of a given subworkflow since User A can send a vistrail to Users B & C who can both make independent edits to a subworkflow and send them back to User A. An issue here is uniquely identifying each of the subworkflows yet possibly maintaining the derivation history. In addition, we have two different types of upgrades that can come into play:		Because we want to allow any user to run a workflow that contains a subworkflow, we save the subworkflows in the vt bundle. This ensures that a user who receives this bundle can run the workflow, assuming they have all the necessary packages installed. At the same time, this means that there can be many different versions of a given subworkflow since User A can send a vistrail to Users B & C who can both make independent edits to a subworkflow and send them back to User A. An issue here is uniquely identifying each of the subworkflows yet possibly maintaining the derivation history. In addition, we have two different types of upgrades that can come into play:

	# User-defined upgrades to the subworkflow (e.g. a user changes a parameter or replaces a module)		# User-defined upgrades to the subworkflow (e.g. a user changes a parameter or replaces a module)
Line 21:		Line 21:
	=== Subworkflows and the registry ===		=== Subworkflows and the registry ===

	Subworkflows are also stored in the registry so that a user might drag them out to use like a normal module. However, for those subworkflows that exist only in an opened vistrail, we do not put them in the registry list (although they must be in the registry); a user must import them into ~~the~~ his own "My Subworkflows" in order to use them in his own work. That said, there may be multiple copies of a subworkflow in the registry, meaning we must have a way to differentiate between them all. This is done via abstraction_uuid and abstraction_origin_uuid annotations in the subworkflow which get used as namespaces and possibly filenames. Obviously, this becomes complicated quickly. With upgraded subworkflows, the different versions must also be managed in the registry. The win with using the registry is that a subworkflow will look like a normal module to most of the VisTrails code.		Subworkflows are also stored in the registry so that a user might drag them out to use like a normal module. However, for those subworkflows that exist only in an opened vistrail, we do not put them in the registry list (although they must be in the registry); a user must import them into his own "My Subworkflows" in order to use them in his own work. That said, there may be multiple copies of a subworkflow in the registry, meaning we must have a way to differentiate between them all. This is done via abstraction_uuid and abstraction_origin_uuid annotations in the subworkflow which get used as namespaces and possibly filenames. Obviously, this becomes complicated quickly. With upgraded subworkflows, the different versions must also be managed in the registry. The win with using the registry is that a subworkflow will look like a normal module to most of the VisTrails code.

	=== Subworkflows and uuids ===		=== Subworkflows and uuids ===

	Subworkflows already use some uuids, but with a uuid for version ids (as in the uuid branch), the issues with identifying which version of a subworkflow (even across multiple files) are largely removed since the uuid will be unique. There still may be some interesting issues about which is the "most recent" version is since the ids are not monotonic.		Subworkflows already use some uuids, but with a uuid for version ids (as in the uuid branch), the issues with identifying which version of a subworkflow (even across multiple files) are largely removed since the uuid will be unique. There still may be some interesting issues about which is the "most recent" version is since the ids are not monotonic.

Dakoop at 23:39, 5 September 2013

2013-09-05T23:39:04Z

← Older revision		Revision as of 23:39, 5 September 2013
Line 1:		Line 1:
			=== Introduction ===

	VisTrails has groups which are "encapsulate by value" entities and subworkflows which are "encapsulate by reference". A group is stored as a workflow with modules and connections (including special InputPort and OutputPort modules that determine which ports are exposed in the group module) while a subworkflow is stored as a version of a workflow in a vistrail. Thus, a change to a group will generate an entirely new workflow that is fully specified while a change to a subworkflow uses change-based provenance to only store the actions between the previous workflow and the new one. Groups are fairly simple and work well to abstract a piece of a workflow but use storage inefficiently. In addition, in their current implementation in VisTrails, they are impossible to edit in place. A user must expand the group, make changes, possibly add back some Input/OutputPort modules, and then group the modules again. Subworkflows solve the storage issue but introduce significant complexity because they must be indexed by the module registry. Subworkflows can exist locally (in a user's .vistrails directory), as part of a distributed package (can use _subworkflows list or add_subworkflow), or as part of a vt bundle. In addition, any reference to a subworkflow in a workflow must both identify the subworkflow and the version to be used.		VisTrails has groups which are "encapsulate by value" entities and subworkflows which are "encapsulate by reference". A group is stored as a workflow with modules and connections (including special InputPort and OutputPort modules that determine which ports are exposed in the group module) while a subworkflow is stored as a version of a workflow in a vistrail. Thus, a change to a group will generate an entirely new workflow that is fully specified while a change to a subworkflow uses change-based provenance to only store the actions between the previous workflow and the new one. Groups are fairly simple and work well to abstract a piece of a workflow but use storage inefficiently. In addition, in their current implementation in VisTrails, they are impossible to edit in place. A user must expand the group, make changes, possibly add back some Input/OutputPort modules, and then group the modules again. Subworkflows solve the storage issue but introduce significant complexity because they must be indexed by the module registry. Subworkflows can exist locally (in a user's .vistrails directory), as part of a distributed package (can use _subworkflows list or add_subworkflow), or as part of a vt bundle. In addition, any reference to a subworkflow in a workflow must both identify the subworkflow and the version to be used.

Dakoop at 23:38, 5 September 2013

2013-09-05T23:38:41Z

← Older revision		Revision as of 23:38, 5 September 2013
Line 16:		Line 16:

	In the first case, the user controls whether an attempt is made to update the subworkflow with a newer version (a leaf in the vistrail) as we do not want to break a workflow by forcing the user to use the new subworkflow. In the second case, the upgrade is required because the subworkflow will not run without the changes. Note that this second upgrade requires a new version in the subworkflow's vistrail as well as a new version in the containing workflow's vistrail because the containing workflow must reference a different "internal version" of the subworkflow.		In the first case, the user controls whether an attempt is made to update the subworkflow with a newer version (a leaf in the vistrail) as we do not want to break a workflow by forcing the user to use the new subworkflow. In the second case, the upgrade is required because the subworkflow will not run without the changes. Note that this second upgrade requires a new version in the subworkflow's vistrail as well as a new version in the containing workflow's vistrail because the containing workflow must reference a different "internal version" of the subworkflow.

			=== Subworkflows and the registry ===

	Subworkflows are also stored in the registry so that a user might drag them out to use like a normal module. However, for those subworkflows that exist only in an opened vistrail, we do not put them in the registry list (although they must be in the registry); a user must import them into the his own "My Subworkflows" in order to use them in his own work. That said, there may be multiple copies of a subworkflow in the registry, meaning we must have a way to differentiate between them all. This is done via abstraction_uuid and abstraction_origin_uuid annotations in the subworkflow which get used as namespaces and possibly filenames. Obviously, this becomes complicated quickly. With upgraded subworkflows, the different versions must also be managed in the registry. The win with using the registry is that a subworkflow will look like a normal module to most of the VisTrails code.		Subworkflows are also stored in the registry so that a user might drag them out to use like a normal module. However, for those subworkflows that exist only in an opened vistrail, we do not put them in the registry list (although they must be in the registry); a user must import them into the his own "My Subworkflows" in order to use them in his own work. That said, there may be multiple copies of a subworkflow in the registry, meaning we must have a way to differentiate between them all. This is done via abstraction_uuid and abstraction_origin_uuid annotations in the subworkflow which get used as namespaces and possibly filenames. Obviously, this becomes complicated quickly. With upgraded subworkflows, the different versions must also be managed in the registry. The win with using the registry is that a subworkflow will look like a normal module to most of the VisTrails code.

			=== Subworkflows and uuids ===

			Subworkflows already use some uuids, but with a uuid for version ids (as in the uuid branch), the issues with identifying which version of a subworkflow (even across multiple files) are largely removed since the uuid will be unique. There still may be some interesting issues about which is the "most recent" version is since the ids are not monotonic.

Dakoop at 23:30, 5 September 2013

2013-09-05T23:30:28Z

← Older revision		Revision as of 23:30, 5 September 2013
Line 10:		Line 10:
	=== Subworkflow versions and upgrades ===		=== Subworkflow versions and upgrades ===

	Because we want subworkflows to		Because we want to allow any user to run a workflow that contains a subworkflow, we save any subworkflows in the vt bundle. This ensures that a user who receives this bundle can run the workflow, assuming their have all the necessary packages installed. At the same time, this means that there can be many different versions of a given subworkflow since User A can send a vistrail to Users B & C who can both make independent edits to a subworkflow and send them back to User A. An issue here is uniquely identifying each of the subworkflows yet possibly maintaining the derivation history. In addition, we have two different types of upgrades that can come into play:

			# User-defined upgrades to the subworkflow (e.g. a user changes a parameter or replaces a module)
			# Package-required upgrades to the subworkflow (e.g. the VTK package installed by a user is newer than the one used to create the original subworkflow)

			In the first case, the user controls whether an attempt is made to update the subworkflow with a newer version (a leaf in the vistrail) as we do not want to break a workflow by forcing the user to use the new subworkflow. In the second case, the upgrade is required because the subworkflow will not run without the changes. Note that this second upgrade requires a new version in the subworkflow's vistrail as well as a new version in the containing workflow's vistrail because the containing workflow must reference a different "internal version" of the subworkflow.

			Subworkflows are also stored in the registry so that a user might drag them out to use like a normal module. However, for those subworkflows that exist only in an opened vistrail, we do not put them in the registry list (although they must be in the registry); a user must import them into the his own "My Subworkflows" in order to use them in his own work. That said, there may be multiple copies of a subworkflow in the registry, meaning we must have a way to differentiate between them all. This is done via abstraction_uuid and abstraction_origin_uuid annotations in the subworkflow which get used as namespaces and possibly filenames. Obviously, this becomes complicated quickly. With upgraded subworkflows, the different versions must also be managed in the registry. The win with using the registry is that a subworkflow will look like a normal module to most of the VisTrails code.

Dakoop at 23:14, 5 September 2013

2013-09-05T23:14:26Z

← Older revision		Revision as of 23:14, 5 September 2013
Line 4:		Line 4:

	One goal is to make editing a group easier. If we could open a window like with editing subworkflows, we could take the editing workflow and use it as the new group. Note, however, that this has a couple issues:		One goal is to make editing a group easier. If we could open a window like with editing subworkflows, we could take the editing workflow and use it as the new group. Note, however, that this has a couple issues:
	1. We must delete any parameters or connections that were attached to ports that were removed (should be able to look at the upgrade code for these routines).		# We must delete any parameters or connections that were attached to ports that were removed (should be able to look at the upgrade code for these routines).
	2. Workflow editing current assumes a vistrail exists which may introduce issues. We have the ability to import workflows from xml so we should be able to create a vistrail from the group's workflow, allow the user to make edits and then pull the currentPipeline back as the group's workflow. Ideally, it would be nice to eventually have a separation between workflow and vistrail such that VisTrails could have a standalone workflow editor with provenance threaded in (this would allow us to use the Provenance SDK).		# Workflow editing current assumes a vistrail exists which may introduce issues. We have the ability to import workflows from xml so we should be able to create a vistrail from the group's workflow, allow the user to make edits and then pull the currentPipeline back as the group's workflow. Ideally, it would be nice to eventually have a separation between workflow and vistrail such that VisTrails could have a standalone workflow editor with provenance threaded in (this would allow us to use the Provenance SDK).
	Another possibility that has been proposed is to make grouping something that is more visual than anything else. In other words, I create a "group" from modules a,b,c by visually collapsing these into a single box but all I write to the vistrail is that certain module ids have been marked as collapsed. The major issue with this is that significant functionality has been built assuming that a group is a tangible entity (e.g. in control flow modules).		Another possibility that has been proposed is to make grouping something that is more visual than anything else. In other words, I create a "group" from modules a,b,c by visually collapsing these into a single box but all I write to the vistrail is that certain module ids have been marked as collapsed. The major issue with this is that significant functionality has been built assuming that a group is a tangible entity (e.g. in control flow modules).

Dakoop: Created page with 'VisTrails has groups which are "encapsulate by value" entities and subworkflows which are "encapsulate by reference". A group is stored as a workflow with modules and connection…'

2013-09-05T23:13:02Z

Created page with 'VisTrails has groups which are "encapsulate by value" entities and subworkflows which are "encapsulate by reference". A group is stored as a workflow with modules and connection…'

New page

VisTrails has groups which are "encapsulate by value" entities and subworkflows which are "encapsulate by reference". A group is stored as a workflow with modules and connections (including special InputPort and OutputPort modules that determine which ports are exposed in the group module) while a subworkflow is stored as a version of a workflow in a vistrail. Thus, a change to a group will generate an entirely new workflow that is fully specified while a change to a subworkflow uses change-based provenance to only store the actions between the previous workflow and the new one. Groups are fairly simple and work well to abstract a piece of a workflow but use storage inefficiently. In addition, in their current implementation in VisTrails, they are impossible to edit in place. A user must expand the group, make changes, possibly add back some Input/OutputPort modules, and then group the modules again. Subworkflows solve the storage issue but introduce significant complexity because they must be indexed by the module registry. Subworkflows can exist locally (in a user's .vistrails directory), as part of a distributed package (can use _subworkflows list or add_subworkflow), or as part of a vt bundle. In addition, any reference to a subworkflow in a workflow must both identify the subworkflow and the version to be used.

=== Group Edits ===

One goal is to make editing a group easier. If we could open a window like with editing subworkflows, we could take the editing workflow and use it as the new group. Note, however, that this has a couple issues:
1. We must delete any parameters or connections that were attached to ports that were removed (should be able to look at the upgrade code for these routines).
2. Workflow editing current assumes a vistrail exists which may introduce issues. We have the ability to import workflows from xml so we should be able to create a vistrail from the group's workflow, allow the user to make edits and then pull the currentPipeline back as the group's workflow. Ideally, it would be nice to eventually have a separation between workflow and vistrail such that VisTrails could have a standalone workflow editor with provenance threaded in (this would allow us to use the Provenance SDK).
Another possibility that has been proposed is to make grouping something that is more visual than anything else. In other words, I create a "group" from modules a,b,c by visually collapsing these into a single box but all I write to the vistrail is that certain module ids have been marked as collapsed. The major issue with this is that significant functionality has been built assuming that a group is a tangible entity (e.g. in control flow modules).

=== Subworkflow versions and upgrades ===

Because we want subworkflows to