Multithreaded-interpreter - Revision history

Remi: /* Issues */

2013-10-31T20:00:45Z

Issues

← Older revision		Revision as of 20:00, 31 October 2013
Line 34:		Line 34:

	== Issues ==		== Issues ==

			=== Things that need work ===
			* Executing multiple modules at once: right now we can only send a single module. Sending multiple modules to be executed at once is only possible by making a group (if all the modules in the group support remote execution).
			* Data movement: currently module input/output is downloaded from clusters and uploaded back for each module (except inside groups). The interpreter should have knowledge of things like files that can't be moved easily (pickling a File object doesn't move the file...)
			* "remotely-executable" semantics: there are modules that we don't need to execute remotely (because they are really fast, like Constant) but shouldn't stop a containing pipeline from being sent out (because they can be executed remotely if needed).

			=== Things to watch out for ===
	* When there are non-parallel modules or tasks, the task system may not run them in the most efficient order. This is because the actual duration of each task is not know. An effort has been made to try to reach the backgroundable tasks first, but as their upstream modules need to be run, finding the most efficient order is difficult. Moreover, using complex strategies to choose a task might make execution slower even though it uses parallelism.		* When there are non-parallel modules or tasks, the task system may not run them in the most efficient order. This is because the actual duration of each task is not know. An effort has been made to try to reach the backgroundable tasks first, but as their upstream modules need to be run, finding the most efficient order is difficult. Moreover, using complex strategies to choose a task might make execution slower even though it uses parallelism.
	* upstream modules are not necessarily run consecutively, this causes issues with packages that wrongly assume that to keep some global state, like matplotlib that creates a Figure then expects only its own upstream modules to be run.		* upstream modules are not necessarily run consecutively, this causes issues with packages that wrongly assume that to keep some global state, like matplotlib [https://www.vistrails.org/ticket/685] that creates a Figure then expects only its own upstream modules to be run.
	* Instances of VisTrails fired up by multiprocessing or IPython load the packages that are listed in the configuration. They won't load other required packages automatically, and they won't ignore unwanted packages (such as visualization packages; VTK takes a long time to load...).		* Instances of VisTrails fired up by multiprocessing or IPython load the packages that are listed in the configuration. They won't load other required packages automatically, and they won't ignore unwanted packages (such as visualization packages; VTK takes a long time to load...).

Remi at 15:51, 31 October 2013

2013-10-31T15:51:14Z

← Older revision		Revision as of 15:51, 31 October 2013
Line 2:		Line 2:

	Currently, support for threads, multiprocessing and IPython has been put in.		Currently, support for threads, multiprocessing and IPython has been put in.

			A general description is available [https://docs.google.com/document/d/167Jhkjt6jXTqwNnkil4whPCKJ8ahKRKPmqbSiSBBq7Q/edit?usp=sharing on Google Docs].

	== Introduction ==		== Introduction ==
Line 167:		Line 169:
	* IPython should provide a serializable job object with the profile and the task UUID		* IPython should provide a serializable job object with the profile and the task UUID
	* resuming a pipeline: suspended module wouldn't run upstream		* resuming a pipeline: suspended module wouldn't run upstream
	* add UI to stop execution if resumable modules are holding it		* add UI to stop execution if only resumable modules are holding it

Remi: /* Integration with job submission */ Meeting summary

2013-09-05T22:11:44Z

Integration with job submission: Meeting summary

← Older revision		Revision as of 22:11, 5 September 2013
Line 161:		Line 161:
	</executionConfiguration>		</executionConfiguration>
	</pre>		</pre>

			=== Integration with job submission ===
			09/05 meeting summary:
			* Store running jobs with a core component somewhere, by serializing the "job object" (currently "queue" object), instead of using the startup.xml ("runningJobsList" key)
			* IPython should provide a serializable job object with the profile and the task UUID
			* resuming a pipeline: suspended module wouldn't run upstream
			* add UI to stop execution if resumable modules are holding it

Remi: /* Execution configuration objects */ Example

2013-07-23T15:21:17Z

Execution configuration objects: Example

← Older revision		Revision as of 15:21, 23 July 2013
Line 136:		Line 136:

	During execution, the ExecutionPreference is passed to the Module in summon() (when the vistrail.module:Module creates a vistrails_module:Module). When the module executes, it uses the parallelization scheme with the highest priority among the ones it supports and that are enabled/available, unless an ExecutionPreference is set in which case it uses that (and fails otherwise).		During execution, the ExecutionPreference is passed to the Module in summon() (when the vistrail.module:Module creates a vistrails_module:Module). When the module executes, it uses the parallelization scheme with the highest priority among the ones it supports and that are enabled/available, unless an ExecutionPreference is set in which case it uses that (and fails otherwise).

			Example of execution configuration file:
			<pre>
			<executionConfiguration version="1.0.3" vtVersion="2.1 beta">
			<!-- This module will only be executed locally -->
			<moduleExecutionPreference module_id="0" preference="-1"/>

			<!-- IPython configuration (with annotation for system-specific settings) -->
			<executionPreference id="0" system="ipython">
			<annotation id="0" key="ipython-profile" value="poly_cluster"/>
			</executionPreference>
			<moduleExecutionPreference module_id="1" preference="0"/>
			<!-- 0,4 means module 4 in group module 0 -->
			<moduleExecutionPreference module_id="0,4" preference="0"/>

			<!-- Another configuration -->
			<executionPreference id="1" system="multiprocessing">
			<annotation id="1" key="processes" value="8"/>
			</executionPreference>
			<moduleExecutionPreference module_id="2" preference="1"/>

			<!-- Unlisted modules will choose automatically at runtime, like they
			currently do -->
			</executionConfiguration>
			</pre>

Remi: Fixed XML comments

2013-07-23T15:18:20Z

Fixed XML comments

← Older revision		Revision as of 15:18, 23 July 2013
Line 48:		Line 48:
	Currently a workflowExec entry in the log looks like this (some parameters have been omitted):		Currently a workflowExec entry in the log looks like this (some parameters have been omitted):

	<workflowExec ip="0.0.0.0" name="ROOT + 32*" tsEnd="2013-07-18 10:38:55">		<pre>
	<annotation id="1" key="__reason__" value="Pipeline Execution" />		<workflowExec ip="0.0.0.0" name="ROOT + 32*" tsEnd="2013-07-18 10:38:55">
	<machine id="1" name="Remram.poly.edu" os="Windows" />		<annotation id="1" key="__reason__" value="Pipeline Execution" />
	<machine id="2" name="Remram.poly.edu" os="Windows" />		<machine id="1" name="Remram.poly.edu" os="Windows" />
	<machine id="3" name="veby" os="Linux" />		<machine id="2" name="Remram.poly.edu" os="Windows" />
	<moduleExec id="1" machine_id="1" moduleId="3" moduleName="PythonSource" />		<machine id="3" name="veby" os="Linux" />
	<moduleExec id="2" machine_id="1" moduleId="0" moduleName="PythonSource" />		<moduleExec id="1" machine_id="1" moduleId="3" moduleName="PythonSource" />
	<moduleExec id="3" machine_id="2" moduleId="3" moduleName="PythonSource">		<moduleExec id="2" machine_id="1" moduleId="0" moduleName="PythonSource" />
	<annotation id="2" key="parallel_execution" value="True" />		<moduleExec id="3" machine_id="2" moduleId="3" moduleName="PythonSource">
	</moduleExec>		<annotation id="2" key="parallel_execution" value="True" />
	<moduleExec id="3" machine_id="3" moduleId="0" moduleName="PythonSource">		</moduleExec>
	<annotation id="3" key="parallel_execution" value="True" />		<moduleExec id="3" machine_id="3" moduleId="0" moduleName="PythonSource">
	</moduleExec>		<annotation id="3" key="parallel_execution" value="True" />
	<moduleExec id="3" machine_id="1" moduleId="2" moduleName="ConcatenateString" />		</moduleExec>
	</workflowExec>		<moduleExec id="3" machine_id="1" moduleId="2" moduleName="ConcatenateString" />
			</workflowExec>
			</pre>

	==== Changes ====		==== Changes ====
Line 71:		Line 73:

	Completely made-up hypothetical example:		Completely made-up hypothetical example:
	<workflowExec name="ROOT + 32*" tsEnd="2013-07-18 10:38:55" reason="Pipeline Execution">		<pre>
	<machine id="1" name="Remram.poly.edu" os="Windows" />		<workflowExec name="ROOT + 32*" tsEnd="2013-07-18 10:38:55" reason="Pipeline Execution">
	<machine id="2" name="win32.cluster.poly.edu" os="Windows" />		<machine id="1" name="Remram.poly.edu" os="Windows" />
	<machine id="3" name="linux.cluster.poly.edu" os="Linux" />		<machine id="2" name="win32.cluster.poly.edu" os="Windows" />
			<machine id="3" name="linux.cluster.poly.edu" os="Linux" />
	<!-- This module executed on IPython via @parallelizable -->
	<moduleExec id="1" machine_id="1" moduleId="0" moduleName="PythonSource">		<!-- This module executed on IPython via @parallelizable -->
	<RemoteExecution scheme="ipython">		<moduleExec id="1" machine_id="1" moduleId="0" moduleName="PythonSource">
	<annotation id="1" key="ipython_profile" value="default" />		<RemoteExecution scheme="ipython">
	<moduleExec id="2" machine_id="2" moduleId="0" moduleName="PythonSource" />		<annotation id="1" key="ipython_profile" value="default" />
	</RemoteExecution>		<moduleExec id="2" machine_id="2" moduleId="0" moduleName="PythonSource" />
	</moduleExec>		</RemoteExecution>
			</moduleExec>
	<!-- This module executed on another IPython profile via @parallelizable -->
	<moduleExec id="3" machine_id="1" moduleId="1" moduleName="PythonSource">		<!-- This module executed on another IPython profile via @parallelizable -->
	<RemoteExecution scheme="ipython">		<moduleExec id="3" machine_id="1" moduleId="1" moduleName="PythonSource">
	<annotation id="1" key="ipython_profile" value="polycluster" />		<RemoteExecution scheme="ipython">
	<moduleExec id="4" machine_id="3" moduleId="1" moduleName="PythonSource" />		<annotation id="1" key="ipython_profile" value="polycluster" />
	</RemoteExecution>		<moduleExec id="4" machine_id="3" moduleId="1" moduleName="PythonSource" />
	</moduleExec>		</RemoteExecution>
			</moduleExec>
	<!-- This module executed locally but made use of an IPython cluster -->
	<moduleExec id="5" machine_id="1" moduleId="2" moduleName="MultiprocessingModule">		<!-- This module executed locally but made use of an IPython cluster -->
	<RemoteExecution scheme="ipython">		<moduleExec id="5" machine_id="1" moduleId="2" moduleName="MultiprocessingModule">
	<annotation id="3" key="ipython_profile" value="default" />		<RemoteExecution scheme="ipython">
	<remoteTask machine_id="2" function="map" count="2" description="Inversing five matrices" />		<annotation id="3" key="ipython_profile" value="default" />
	<remoteTask machine_id="3" function="map" count="3" description="Inversing five matrices" />		<remoteTask machine_id="2" function="map" count="2" description="Inversing five matrices" />
	</RemoteExecution>		<remoteTask machine_id="3" function="map" count="3" description="Inversing five matrices" />
	<RemoteExecution scheme="ipython">		</RemoteExecution>
	<annotation id="4" key="ipython_profile" value="polycluster" />		<RemoteExecution scheme="ipython">
	<remoteTask machine_id="3" function="apply" count="1" description="Graphviz" />		<annotation id="4" key="ipython_profile" value="polycluster" />
	</RemoteExecution>		<remoteTask machine_id="3" function="apply" count="1" description="Graphviz" />
	</moduleExec>		</RemoteExecution>
			</moduleExec>
	<!-- This module only used the local machine -->
	<moduleExec id="6" machine_id="1" moduleId="3" moduleName="ConcatenateString" />		<!-- This module only used the local machine -->
	</workflowExec>		<moduleExec id="6" machine_id="1" moduleId="3" moduleName="ConcatenateString" />
			</workflowExec>
			</pre>

	=== Choosing per-module execution target ===		=== Choosing per-module execution target ===

Remi: /* Changes */ Comments

2013-07-23T15:14:03Z

Changes: Comments

← Older revision		Revision as of 15:14, 23 July 2013
Line 76:		Line 76:
	<machine id="3" name="linux.cluster.poly.edu" os="Linux" />		<machine id="3" name="linux.cluster.poly.edu" os="Linux" />

			<!-- This module executed on IPython via @parallelizable -->
	<moduleExec id="1" machine_id="1" moduleId="0" moduleName="PythonSource">		<moduleExec id="1" machine_id="1" moduleId="0" moduleName="PythonSource">
	<RemoteExecution scheme="ipython">		<RemoteExecution scheme="ipython">
Line 83:		Line 84:
	</moduleExec>		</moduleExec>

			<!-- This module executed on another IPython profile via @parallelizable -->
	<moduleExec id="3" machine_id="1" moduleId="1" moduleName="PythonSource">		<moduleExec id="3" machine_id="1" moduleId="1" moduleName="PythonSource">
	<RemoteExecution scheme="ipython">		<RemoteExecution scheme="ipython">
Line 90:		Line 92:
	</moduleExec>		</moduleExec>

			<!-- This module executed locally but made use of an IPython cluster -->
	<moduleExec id="5" machine_id="1" moduleId="2" moduleName="MultiprocessingModule">		<moduleExec id="5" machine_id="1" moduleId="2" moduleName="MultiprocessingModule">
	<RemoteExecution scheme="ipython">		<RemoteExecution scheme="ipython">
Line 97:		Line 100:
	</RemoteExecution>		</RemoteExecution>
	<RemoteExecution scheme="ipython">		<RemoteExecution scheme="ipython">
	<annotation id="4" key="ipython_profile" value="~~default~~" />		<annotation id="4" key="ipython_profile" value="polycluster" />
	<remoteTask machine_id="3" function="apply" count="1" description="Graphviz" />		<remoteTask machine_id="3" function="apply" count="1" description="Graphviz" />
	</RemoteExecution>		</RemoteExecution>
	</moduleExec>		</moduleExec>

			<!-- This module only used the local machine -->
	<moduleExec id="6" machine_id="1" moduleId="3" moduleName="ConcatenateString" />		<moduleExec id="6" machine_id="1" moduleId="3" moduleName="ConcatenateString" />
	</workflowExec>		</workflowExec>

Remi: /* Choosing per-module execution target */ Execution configuration

2013-07-22T19:29:04Z

Choosing per-module execution target: Execution configuration

← Older revision		Revision as of 19:29, 22 July 2013
Line 120:		Line 120:
	** OR: annotate modules (but this interferes with versioning)		** OR: annotate modules (but this interferes with versioning)
	** OR: use a port (interferes with versioning)		** OR: use a port (interferes with versioning)

			==== Execution configuration objects ====
			* vistrails_module:Module has a static supported_execution:SupportedExecution, initialized through the @parallelizable class decorator. This only contains what systems are supported.
			* ModuleDescriptor has the same supported_execution, extracted from the Module. It is used by the UI to show choices to the user.
			* vistrails.module:Module has an execution_preference:ExecutionPreference that is set by the user. This contains one supported system plus system-specific settings (cluster name, target address, ...) as annotations. Different modules can share an ExecutionPreference.
			* A whole pipeline's configuration is represented by ExecutionConfiguration. It contains a mapping from a list of module ids (string format: "{group_id},{group_id},{module_id}" to identify nested modules) to an ExecutionPreference.

			During execution, the ExecutionPreference is passed to the Module in summon() (when the vistrail.module:Module creates a vistrails_module:Module). When the module executes, it uses the parallelization scheme with the highest priority among the ones it supports and that are enabled/available, unless an ExecutionPreference is set in which case it uses that (and fails otherwise).

Remi: /* Current format */ typo

2013-07-22T18:25:53Z

Current format: typo

← Older revision		Revision as of 18:25, 22 July 2013
Line 46:		Line 46:

	==== Current format ====		==== Current format ====
	Currently a workflowExec entry in the log looks like this (some ~~parameter~~ have been omitted):		Currently a workflowExec entry in the log looks like this (some parameters have been omitted):

	<workflowExec ip="0.0.0.0" name="ROOT + 32*" tsEnd="2013-07-18 10:38:55">		<workflowExec ip="0.0.0.0" name="ROOT + 32*" tsEnd="2013-07-18 10:38:55">

Remi: Added a package loading issue

2013-07-19T21:37:04Z

Added a package loading issue

← Older revision		Revision as of 21:37, 19 July 2013
Line 34:		Line 34:
	* When there are non-parallel modules or tasks, the task system may not run them in the most efficient order. This is because the actual duration of each task is not know. An effort has been made to try to reach the backgroundable tasks first, but as their upstream modules need to be run, finding the most efficient order is difficult. Moreover, using complex strategies to choose a task might make execution slower even though it uses parallelism.		* When there are non-parallel modules or tasks, the task system may not run them in the most efficient order. This is because the actual duration of each task is not know. An effort has been made to try to reach the backgroundable tasks first, but as their upstream modules need to be run, finding the most efficient order is difficult. Moreover, using complex strategies to choose a task might make execution slower even though it uses parallelism.
	* upstream modules are not necessarily run consecutively, this causes issues with packages that wrongly assume that to keep some global state, like matplotlib that creates a Figure then expects only its own upstream modules to be run.		* upstream modules are not necessarily run consecutively, this causes issues with packages that wrongly assume that to keep some global state, like matplotlib that creates a Figure then expects only its own upstream modules to be run.
			* Instances of VisTrails fired up by multiprocessing or IPython load the packages that are listed in the configuration. They won't load other required packages automatically, and they won't ignore unwanted packages (such as visualization packages; VTK takes a long time to load...).

	== Current work ==		== Current work ==

Remi at 16:12, 19 July 2013

2013-07-19T16:12:27Z

← Older revision		Revision as of 16:12, 19 July 2013
Line 109:		Line 109:
	Instead, it would make sense to be able to select a target for a particular module among what it supports. This means:		Instead, it would make sense to be able to select a target for a particular module among what it supports. This means:
	* The UI needs to know what a module support; hence it should no longer just be an execution thing.		* The UI needs to know what a module support; hence it should no longer just be an execution thing.
	* We could keep the current class-decorator, that we would extract to the ModuleDescriptor, or add a field (like _input_ports) and/or a parameter to ModuleRegistry#add_module().		** We could keep the current class-decorator, that we would extract to the ModuleDescriptor
			** or add a field (like _input_ports) and/or a parameter to ModuleRegistry#add_module().
			* The interpreter needs to know what was chosen in the UI
			** As simple as setting a field in Module#summon()
	* Needs actual UI support to choose the scheme, and the specific parameters for that scheme		* Needs actual UI support to choose the scheme, and the specific parameters for that scheme
	* Probably needs to allow VisTrails to connect to several IPython clusters at once, since we can now assign different modules to different clusters.		* Probably needs to allow VisTrails to connect to several IPython clusters at once, since we can now assign different modules to different clusters.