NYU HPC Access Instructions
Accessing the NYU HPC Cluster
1. Log into the main HPC node:
2. From the HPC node, log into the Hadoop cluster:
You will be using a set of commands, and it will save you some time to first create aliases for them. Once on "dumbo", run the following commands on your terminal:
bash alias hfs='/usr/bin/hadoop fs ' export HAS=/opt/cloudera/parcels/CDH-5.4.5-1.cdh5.4.5.p0.7/jars export HSJ=hadoop-streaming-2.6.0-cdh5.4.5.jar alias hjs='/usr/bin/hadoop jar $HAS/$HSJ'
To be able to re-use these aliases every time you login to dumbo, append the following lines to the end of your .bashrc file: alias hfs='/usr/bin/hadoop fs ' export HAS=/opt/cloudera/parcels/CDH-5.4.5-1.cdh5.4.5.p0.7/jars export HSJ=hadoop-streaming-2.6.0-cdh5.4.5.jar alias hjs='/usr/bin/hadoop jar $HAS/$HSJ'
%% Note: you should not have any spaces around "="!
If you have bash as your default shell, do
This will create the aliases. If you have tcsh as your default shell, just invoke bash -- it will automatically read the .bashrc file and create the aliases.
Here are some common commands: hfs %% See available commands. hfs -help %% more command details. hfs -ls [<path>] %% List files hfs -cp <src> <dst> %% Copy stuff hfs -mkdir <path> %% Create path hfs -rm <path> %% remove a file hfs -chmod <path> %% Modify permissions. hfs -chown <path> %% Modify owner.
Some remote access commands: hfs -cat <src> %% Cat contents to stdout. hfs -copyFromLocal <localsrc> <dst> %% Copy stuff hfs -copyToLocal <src> <localdst> %% Copy stuff
Using Hadoop Streaming
- Hadoop streaming allows the use any program written in any language for mapreduce operations.
- You can use the "hjs" alias you created to run Hadoop Streaming
To run the example I provided, do the following:
1) Copy the directory containing the Python files and input data to dumbo. You will first need to "scp" from your machine to the hpc node, and them from the hpc node to dumbo. Assuming the directory is called /Users/julianafreire/MRExample
scp -r /Users/julianafreire/MRExample email@example.com:
Then, from the hpc node:
scp -r MRExample dumbo
- Remember to replace your_netid with your actual netid!
2) From dumbo, you will now copy the data file to HDFS
hfs -copyFromLocal /home/you_netid/MRExample/wikipedia.txt wikipedia.txt
3) Check if the file is on HDFS
4) Now, to run the job, make sure you are on the right directory
cd /home/your_netid/MRExample hjs -file pmap.py -mapper pmap.py -file pred.py -reducer pred.py -input /user/your_netid/wikipedia.txt -output /user/your_netid/wikipedia.output
5) The outputs of this job are now in HDFS, in the directory /user/your_netid/wikipedia.output. To list the output files:
hfs -ls /user/jf1870/wikipedia.output
You can also inspect the content of the files:
hfs -cat wikipedia.output/*
If you'd like to copy the files over to your local directory:
hfs -get /user/jf1870/wikipedia.output output
This will copy the outputs to the local directory "output" on dumbo
- Spark allow you to write and run applications quickly in Java, Scala, Python and R
- You can either use Spark interactive shell or Spark submission tool
To run Spark interactive shell (Scala or Python):
1) Login to dumbo
2) Execute one of the following: spark-shell (to run applications in Scala)
pyspark (to run applications in Python)
If you want to access your files stored on HDFS, use the following URL as filename in Spark hdfs://babar.es.its.nyu.edu:8020/user/<your_net_id>/<your_files> (the hdfs:// URL must be absolutely correct, otherwise you won't be able to access file from HDFS)
To submit job to Spark:
1) Login to dumbo
2) Execute spark-submit --num-executors <10-100> <your_python_script> <your_script_arguments>
DUMBO cluster has 100 executors. Feel free to choose any number of executors for your submission. The bigger the faster. However if many people submit Spark job at the same time, performance will be downgraded.
Spark word count example:
Without streaming: https://github.com/apache/spark/blob/master/examples/src/main/python/wordcount.py With streaming: https://github.com/apache/spark/blob/master/examples/src/main/python/streaming/hdfs_wordcount.py
Spark streaming is not the same as Hadoop streaming: in contrast with Hadoop, you can originally run Python/R/Java/Scala script in Spark. The difference is that Spark Streaming provide streaming processing of live data stream.
1) Submitting application to Spark: http://spark.apache.org/docs/latest/submitting-applications.html 2) Data transformation: http://spark.apache.org/docs/latest/programming-guide.html#transformations