Hadoop Package - Revision history

Remi: github.org -> github.com

2014-02-26T15:01:40Z

github.org -> github.com

← Older revision		Revision as of 15:01, 26 February 2014
Line 10:		Line 10:
	Install vistrails from source:		Install vistrails from source:
	git clone http://vistrails.org/git/vistrails.git		git clone http://vistrails.org/git/vistrails.git
	git clone https://github.~~org~~/rexissimus/BatchQ-PBS -b remoteq		git clone https://github.com/rexissimus/BatchQ-PBS -b remoteq
	cp -r BatchQ-PBS/remoteq vistrails/		cp -r BatchQ-PBS/remoteq vistrails/
	cd vistrails		cd vistrails

Tommy: /* Machine */

2014-02-25T15:53:09Z

Machine

← Older revision		Revision as of 15:53, 25 February 2014
Line 22:		Line 22:

	==== Machine ====		==== Machine ====
	Represents a remote machine running SSH~~. If you specify the machine information in the RemoteQ package configuration you do not need to use a Machine in your workflow~~.		Represents a remote machine running SSH.
	* server - the server url		* server - the server url
	* username - the remote server username, default is your local username		* username - the remote server username, default is your local username
	* password - your password, connect a PasswordDialog to here		* password - your password, connect a PasswordDialog to here
	* port - the remote ssh port, set to 0 if using an ssh tunnel		* port - the remote ssh port, set to 0 if using an ssh tunnel

			If you specify the machine information in the RemoteQ package configuration you do not need to use a Machine in your workflow:

			{\| border="1"
			\| server \|\| server url or alias
			\|-
			\| username \|\| your username on the server, leave blank to use your local username.
			\|-
			\| port \|\| SSH port on the server, leave blank if using default, use 0 if using an alias.
			\|-
			\| password \|\| Do you need to specify a password to log in.
			\|-
			\| defaultFS \|\| The default filesystem path to use. Example: "s3n://<yourusername>/"
			\|-
			\| uris \|\| A list of path#alias tuples separated by ";". Using <alias> in the workflow will replace it with <path> before being executed by hadoop.
			\|}

	==== HadoopStreaming ====		==== HadoopStreaming ====

Tommy: /* Modules used with the hadoop package */

2014-02-25T15:47:11Z

Modules used with the hadoop package

← Older revision		Revision as of 15:47, 25 February 2014
Line 19:		Line 19:
	The BatchQ library used by the RemoteQ package does not support Windows. But you should be able to run Linux in a [http://www.virtualbox.org virtual machine] and install vistrails there.		The BatchQ library used by the RemoteQ package does not support Windows. But you should be able to run Linux in a [http://www.virtualbox.org virtual machine] and install vistrails there.

	== ~~Modules used with~~ the hadoop package ==		== Using the hadoop package ==

	==== Machine ====		==== Machine ====

Tommy: /* Machine */

2014-02-25T15:46:21Z

Machine

← Older revision		Revision as of 15:46, 25 February 2014
Line 22:		Line 22:

	==== Machine ====		==== Machine ====
	Represents a remote machine running SSH.		Represents a remote machine running SSH. If you specify the machine information in the RemoteQ package configuration you do not need to use a Machine in your workflow.
	* server - the server url		* server - the server url
	* username - the remote server username, default is your local username		* username - the remote server username, default is your local username

Tommy: /* Using the cluster at gray02.poly.edu */

2014-02-25T15:44:38Z

Using the cluster at gray02.poly.edu

← Older revision		Revision as of 15:44, 25 February 2014
Line 98:		Line 98:
	\| uris \|\| hdfs:///user/tommy/wikitext-big-notitle.csv#wikitext-big-notitle.csv		\| uris \|\| hdfs:///user/tommy/wikitext-big-notitle.csv#wikitext-big-notitle.csv
	\|}		\|}
	~~[[image:hadoop_workflow.png]]~~

	== Using Amazon AWS ==		== Using Amazon AWS ==

Tommy: /* Using Amazon AWS */

2014-02-25T15:43:40Z

Using Amazon AWS

← Older revision		Revision as of 15:43, 25 February 2014
Line 102:		Line 102:
	== Using Amazon AWS ==		== Using Amazon AWS ==

	First do ~~the~~ [[AWS_Setup]].		First do [[AWS_Setup]].

	AWS uses "*.pem" key files for access. Make sure you have one, then edit ~/.ssh/config and add		AWS uses "*.pem" key files for access. Make sure you have one, then edit ~/.ssh/config and add
Line 127:		Line 127:
	Change defaultFS to your s3 bucket		Change defaultFS to your s3 bucket

	An example ~~file is available at vistrails~~-~~hadoop/aws~~.vt. ~~It contains a working hadoop~~ workflow. ~~Change~~ the ~~S3 bucket instances~~ to ~~point~~ to ~~your bucket~~ and ~~execute~~. ~~When~~ it ~~finishes you should see~~ the ~~same result as in~~ the ~~[[#Example\|Example above]]~~.		== Example ==
			Lets run a basic example with a mapper that returns info about the machine it was executed on.


			Open hadoop-nodeinfo.vt. Execute the workflow. The workflow will halt while waiting for the job to finish. Pressing cancel will detach the running job an add it to the Job Monitor. The status of the execution can be checked by right-clicking HadoopStreaming and (for hadoop) selecting "View Standard error". Click "Update" to refresh the view.
			The job can be resumed by re-executing the workflow in vistrails. Once it completes the spreadsheet will list info about the 20 lines processed by the mapper. (usually the same)

Tommy: /* Example */

2014-02-25T15:41:00Z

Example

← Older revision		Revision as of 15:41, 25 February 2014
Line 69:		Line 69:
	* Delete the job in the job monitor by selecting it an pressing "Del"		* Delete the job in the job monitor by selecting it an pressing "Del"

	== ~~Example ==~~		== Using the cluster at gray02.poly.edu ==
	~~Lets try using~~ gray02.poly.edu to ~~run basic example with~~ a ~~mapper that returns info about~~ the ~~machine it was executed on~~.		If you are outside the poly network you need to use a ssh tunnel to get through the firewall.

			Add this to ~/.ssh/config:
	~~You will need an account on~~ vgchead.		Host vgctunnel
	In a ~~terminal run~~:		HostName vgchead.poly.edu
			LocalForward 8101 gray02.poly.edu:22

			Host gray02
			HostName localhost
			Port 8101
			Set up a tunnel to gray02 by running:
	ssh vgctunnel		ssh vgctunnel
	~~Enter your password and keep the window open.~~
	~~Open vistrails-hadoop/example_nodeinfo.vt. It contains a working hadoop workflow.~~

	~~Enter~~ the machine info by going to Preferences->Module Packages, select RemoteQ and click "configure...". Enter this in the configuration:		In vistrails, enter the machine info by going to Preferences->Module Packages, select RemoteQ and click "configure...". Enter this in the configuration:
	{\| border="1"		{\| border="1"
	\| server \|\| gray02		\| server \|\| gray02
Line 95:		Line 99:
	\|}		\|}
	[[image:hadoop_workflow.png]]		[[image:hadoop_workflow.png]]

	Execute the workflow. The workflow will halt while waiting for the job to finish. Pressing cancel will detach the running job an add it to the Job Monitor. The status of the execution can be checked by right-clicking HadoopStreaming and (for hadoop) selecting "View Standard error".
	~~The job can be resumed by re-executing the workflow in vistrails. Once it completes the spreadsheet will list info about the 20 lines processed by the mapper. (usually the same)~~

	~~===== Using the cluster at gray02.poly.edu =====~~
	~~If you are outside the poly network you need to use a ssh tunnel to get through the firewall.~~

	~~Add this to ~/.ssh/config:~~
	~~Host vgctunnel~~
	~~HostName vgchead.poly.edu~~
	~~LocalForward 8101 gray02.poly.edu:22~~

	~~Host gray02~~
	~~HostName localhost~~
	~~Port 8101~~
	~~Set up a tunnel to gray02 by running:~~
	~~ssh vgctunnel~~


	== Using Amazon AWS ==		== Using Amazon AWS ==

Tommy: /* Using Amazon AWS */

2014-02-25T15:37:03Z

Using Amazon AWS

← Older revision		Revision as of 15:37, 25 February 2014
Line 98:		Line 98:
	Execute the workflow. The workflow will halt while waiting for the job to finish. Pressing cancel will detach the running job an add it to the Job Monitor. The status of the execution can be checked by right-clicking HadoopStreaming and (for hadoop) selecting "View Standard error".		Execute the workflow. The workflow will halt while waiting for the job to finish. Pressing cancel will detach the running job an add it to the Job Monitor. The status of the execution can be checked by right-clicking HadoopStreaming and (for hadoop) selecting "View Standard error".
	The job can be resumed by re-executing the workflow in vistrails. Once it completes the spreadsheet will list info about the 20 lines processed by the mapper. (usually the same)		The job can be resumed by re-executing the workflow in vistrails. Once it completes the spreadsheet will list info about the 20 lines processed by the mapper. (usually the same)

			===== Using the cluster at gray02.poly.edu =====
			If you are outside the poly network you need to use a ssh tunnel to get through the firewall.

			Add this to ~/.ssh/config:
			Host vgctunnel
			HostName vgchead.poly.edu
			LocalForward 8101 gray02.poly.edu:22

			Host gray02
			HostName localhost
			Port 8101
			Set up a tunnel to gray02 by running:
			ssh vgctunnel


	== Using Amazon AWS ==		== Using Amazon AWS ==

Tommy: /* Modules used with the hadoop package */

2014-02-25T15:35:43Z

Modules used with the hadoop package

← Older revision		Revision as of 15:35, 25 February 2014
Line 21:		Line 21:
	== Modules used with the hadoop package ==		== Modules used with the hadoop package ==

	==== ~~Dialogs.PasswordDialog ====~~		==== Machine ====
	~~Used to specify a password to the remote machine~~

	~~==== Remote PBS.~~Machine ====
	Represents a remote machine running SSH.		Represents a remote machine running SSH.
	* server - the server url		* server - the server url
	* username - the remote server username, default is your local username		* username - the remote server username, default is your local username
	* password - your password, connect ~~the~~ PasswordDialog to here		* password - your password, connect a PasswordDialog to here
	* port - the remote ssh port, set to 0 if using an ssh tunnel		* port - the remote ssh port, set to 0 if using an ssh tunnel

	~~===== Connecting to the Poly cluster through vgchead =====~~
	~~The hadoop job submitter runs on gray02.poly.edu. If you are outside the poly network you need to use a ssh tunnel to get through the firewall.~~

	~~Add this to ~/.ssh/config:~~
	~~Host vgctunnel~~
	~~HostName vgchead.poly.edu~~
	~~LocalForward 8101 gray02.poly.edu:22~~

	~~Host gray02~~
	~~HostName localhost~~
	~~Port 8101~~
	~~ForwardX11 yes~~
	~~Set up a tunnel to gray02 by running:~~
	~~ssh vgctunnel~~
	~~In vistrails, create a Machine module with host=gray02 and port=0. Now you have a connection that can be used by the hadoop package~~

	==== HadoopStreaming ====		==== HadoopStreaming ====

Tommy: /* Installation */

2014-02-25T15:30:47Z

Installation

← Older revision		Revision as of 15:30, 25 February 2014
Line 2:		Line 2:

	== Installation ==		== Installation ==
	=== ~~Install vistrails~~ ===		=== Mac ===
	~~Get~~ vistrails ~~using git and check out a version supporting~~ the hadoop package:		This binary version of vistrails has the hadoop package preinstalled:

			http://vgc.poly.edu/files/tommy/vistrails-mac-10.6-master-2014-02-25.dmg

			=== Linux ===
			Install vistrails from source:
	git clone http://vistrails.org/git/vistrails.git		git clone http://vistrails.org/git/vistrails.git
			git clone https://github.org/rexissimus/BatchQ-PBS -b remoteq
			cp -r BatchQ-PBS/remoteq vistrails/
	cd vistrails		cd vistrails
	~~git checkout 976255974f2b206f030b2436a5f10286844645b0~~		python vistrails/run.py
	~~On mac you can link this to the binary using~~ vistrails/~~scripts/mac_update_bin.sh~~.		The first time vistrails is started it will download and install all the dependencies.

	~~=== Install BatchQ-PBS~~ and the ~~RemotePBS package ===~~
	~~This python package is used for communication over ssh. Get it with:~~
	~~git clone https://github.com/rexissimus/BatchQ-PBS~~
	~~Copy BatchQ-PBS/batchq to your vistrails python installations site-packages folder.~~

	~~Copy BatchQ-PBS/batchq/contrib/vistrails/RemotePBS to ~/~~.~~vistrails/userpackages/~~

	=== ~~Install the hadoop package~~ ===		=== Windows ===
	~~git clone git~~://~~vgc.poly~~.~~edu:src/vistrails-hadoop~~.~~git~~		The BatchQ library used by the RemoteQ package does not support Windows. But you should be able to run Linux in a [http://www.virtualbox.org virtual machine] and install vistrails there.
	~~ln -s `pwd`/~~vistrails~~-hadoop/hadoop ~/~~.~~vistrails/userpackages/hadoop~~

	== Modules used with the hadoop package ==		== Modules used with the hadoop package ==