= DS-GA 1004- Big Data: Tentative Schedule -- ''subject to change'' =

* Course Web page: http://vgc.poly.edu/~juliana/courses/BigData2016

* Instructors:
** Professor Juliana Freire (http://vgc.poly.edu/~juliana)
** Dr. Erin C Carson
** Dr. Nicholas Knight

* TAs:
** Yuan Feng
** Kevin Ye

* Lecture: Mondays, 4:55pm-7:35pm at Silver 207

* Some classes will include a lab session, please always ''bring your laptop''.

= News =

* 1/25/2016: Amazon has kindly donated time on AWS for all the student in this class. You must signup for the AWS Educate program, see http://www.vistrails.org/index.php/AWS_Setup
* 1/25/2016: An HPC account has been created for you. You will need this account for in-class exercises and homework assignments. See [[NYU HPC Access Instructions]]

== Week 1 - Jan 25: Course Overview ==

* '''Lecture notes:''' http://vgc.poly.edu/~juliana/courses/BigData2016/Lectures/course-overview.pdf
*''' Lab:''' Computing infrastructure for the course
* '''Reading:''' Chapter 1 of Mining of Massive Data Sets (version 1.1)
* '''Course survey:''' https://docs.google.com/forms/d/1LTiJwkDVvp0cF62Fw_d9Y86US5LCkorRUIQtV2T8KWE/viewform?usp=send_form

== Week 2 - Feb 1: The evolution of Data Management and introduction to Big Data; Introduction to Databases and Relational Model ==

* '''Lecture notes:'''
** http://vgc.poly.edu/~juliana/courses/BigData2016/Lectures/data-management-evolution.pdf
** http://vgc.poly.edu/~juliana/courses/BigData2016/Lectures/intro-to-db.pdf
* '''Lab:''' getting started with MySQL
* '''Required Reading:'''
** Chapter 1 of Mining of Massive Data Analysis
* '''Suggested Reading:'''
** [http://philip.greenspun.com/sql/introduction.html Greenspun's SQL for Web Nerds Intro]
** [http://philip.greenspun.com/sql/data-modeling.html SQL/Nerds Modeling (parts)]
** [https://docs.google.com/file/d/0B7lNUaak0bK1NDBWZU5XTmItdGc/edit History Repeats Itself: Sensible and NonsenSQL Aspects of the NoSQL Hoopla], by C. Mohan, EDBT 2013

== Week 3 - Feb 8: Introduction to Databases, Relational Model and SQL (cont.) ==

*''' Lecture notes:'''
** http://vgc.poly.edu/~juliana/courses/BigData2016/Lectures/relational-algebra.pdf
** http://vgc.poly.edu/~juliana/courses/BigData2016/Lectures/sql.pdf
* '''Lab:''' SQL
* '''Programming assignment:''' Using SQL for data analysis and cleaning (check NYU Classes)

== Week 4 - Feb 15: Holiday ==

= Transparency and Reproducibility (1 week) =

== Week 5 - Feb 22: Data Exploration and Reproducibility ==

* '''Lecture notes:''' http://vgc.poly.edu/~juliana/courses/BigData2016/Lectures/reproducibility-provenance.pdf
* '''Lab:''' Hands-on git and github (see NYU Classes). You will need to submit your work for this lab!

= Big Data Foundations and Infrastructure (3 weeks) =

== Week 6 - Feb 29: Introduction to Map Reduce ==

*''' Lecture notes:''' http://vgc.poly.edu/~juliana/courses/BigData2016/Lectures/mapreduce-intro.pdf
* '''Lab:''' Hands-on Hadoop (local and AWS)
* Quiz 1 (Map Reduce) -- check http://www.newgradiance.com/services (juliana_freire: Ch. 2: Map-Reduce)
** Quiz is due on 2016-03-14 12:00 PM EST

== Week 7 - March 7: MapReduce Algorithm Design Patterns ==

*''' Lecture notes:'''
** http://vgc.poly.edu/~juliana/courses/BigData2016/Lectures/mapreduce-recap.pdf
** http://vgc.poly.edu/~juliana/courses/BigData2016/Lectures/mapreduce-algo-design-patterns.pdf
** http://vgc.poly.edu/~juliana/courses/BigData2016/Lectures/mapreduce-algo-design-relations.pdf
* '''Lab:''' Hands-on Hadoop (HPC)
* '''Programming assignment:''' Map Reduce (check NYU Classes)
* '''Readings''':
** Data-Intensive Text Processing with MapReduce, Chapters 1 and 2
** Data-Intensive Text Processing with MapReduce (Jan 27, 2013), Chapter 6 -- Processing Relational Data (this chapter appears in the 2013 version of the textbook -- http://lintool.github.io/MapReduceAlgorithms/ed1n/MapReduce-algorithms.pdf)

== Week 8-- March 14th: Spring Break ==

== Week 9- March 21st: Parallel Databases vs MapReduce; Storage Solutions; Introduction to SPARK==

*''' Lecture notes:'''
** http://vgc.poly.edu/~juliana/courses/BigData2016/Lectures/paralleldb-vs-hadoop.pdf
** http://vgc.poly.edu/~juliana/courses/BigData2016/Lectures/hive-pig.pdf
* '''Lab:''' NoSQL
* Assignment: Hands-on Map-Reduce (see NYU Classes)
* '''Readings''':
** Benchmark DBMS vs MapReduce (2009): http://database.cs.brown.edu/sigmod09/benchmarks-sigmod09.pdf
** MapReduce: A Flexible Data Processing Tool: http://cacm.acm.org/magazines/2010/1/55744-mapreduce-a-flexible-data-processing-tool/fulltext
** Hive - A Warehousing Solution Over a Map-Reduce Framework. http://dl.acm.org/citation.cfm?id=1687609
** Pig Latin: A Not-So-Foreign Language for Data Processing. http://dl.acm.org/citation.cfm?id=1376726
* '''Additional Suggested reading:'''
** BigTable: http://fcoffice.googlecode.com/svn/%E4%B9%A6%E7%B1%8D/bigtable-osdi06.pdf
** Spark: Cluster Computing with Working Sets. http://static.usenix.org/legacy/events/hotcloud10/tech/full_papers/Zaharia.pdf

= Big Data Algorithms, Mining Techniques, and Visualization (6 weeks) =

== Week 10 - March 28th: Finding similar items ==

* Lecture notes:
** http://vgc.poly.edu/~juliana/courses/BigData2016/Lectures/similarity.pdf

* Reading: Chapter 3 [http://vgc.poly.edu/~juliana/courses/BigData2016/Textbooks/ullman-book-v1.1-mining-massive-data.pdf Mining of Massive Datasets]

* Homework Assignment
** See quizzes on [http://www.newgradiance.com Gradiance] -- Distance measures and document similarity.

== Week 11 - April 4th: Association Rules ==

* Lecture notes:
** http://vgc.poly.edu/~juliana/courses/BigData2016/Lectures/association-rules.pdf

* Reading: Chapter 6 [http://vgc.poly.edu/~juliana/courses/MassiveDataAnalysis2014/Textbooks/ullman-book-v1.1-mining-massive-data.pdf Mining of Massive Datasets]

* Suggested additional reading:
**Fast algorithms for mining association rules, Agrawal and Srikant, VLDB 1994.
**Data Mining Concepts and Techniques, Jiawei Han and Micheline Kamber, Morgan Kaufmann
**Dynamic Itemset Counting and Implication Rules for Market Basket Data. Brin et al., SIGMOD 1997. http://www-db.stanford.edu/~sergey/dic.html

* Homework Assignment
** See quizes on [http://www.newgradiance.com Gradiance] -- Distance measures and document similarity.

== Week 12 - April 11th: Visualization and Spatio-Temporal Data -- Invited lecture by Dr. Harish Doraiswamy (NYU CDS) ==

* Lecture notes:
** http://vgc.poly.edu/~juliana/courses/BigData2016/Lectures/SpatialQP.pdf

* Lab: Using Amazon AWS to analyze and visualize taxi data
** https://github.com/ViDA-NYU/aws_taxi

== Week 13 - April 18th: Data Cleaning - Invited lecture by Dr. Divesh Srivastava, AT&T Research ==

== Week 14 - April 25th: Graph Analysis ==

* '''Lecture notes:''' http://vgc.poly.edu/~juliana/courses/BigData2015/Lectures/graph-algos.pdf

* Required Reading: Data-Intensive Text Processing with MapReduce. Chapters 5 -- Graph Algorithms

== Week 15 - May 2: TBD ==

== Week 16 - May 9: Final Exam ==

== Week 17 - May 16: Project Presentations ==

Course: Big Data 2016

2016-03-20T17:47:07Z

Course: Big Data 2016

2016-01-23T22:01:34Z

Juliana:

= DS-GA 1004- Big Data: Tentative Schedule -- ''subject to change'' =

* Course Web page: http://vgc.poly.edu/~juliana/courses/BigData2016

* Instructors:
** Professor Juliana Freire (http://vgc.poly.edu/~juliana)
** Dr. Erin C Carson
** Dr. Nicholas Knight

* TAs:
** Yuan Feng
** Kevin Ye

* Lecture: Mondays, 4:55pm-7:35pm at 19 University Pl., room 102.

* Some classes will include a lab session, please always ''bring your laptop''.

= News =

* 1/25/2016: Amazon has kindly donated time on AWS for all the student in this class. You must signup for the AWS Educate program, see http://www.vistrails.org/index.php/AWS_Setup
* 1/25/2016: An HPC account has been created for you. You will need this account for in-class exercises and homework assignments. See [[NYU HPC Access Instructions]]

== Week 1 - Jan 25: Course Overview ==

* '''Lecture notes:''' http://vgc.poly.edu/~juliana/courses/BigData2016/Lectures/course-overview.pdf
*''' Lab:''' Computing infrastructure for the course
* '''Reading:''' Chapter 1 of Mining of Massive Data Sets (version 1.1)
* '''Course survey:''' https://docs.google.com/forms/d/1LTiJwkDVvp0cF62Fw_d9Y86US5LCkorRUIQtV2T8KWE/viewform?usp=send_form

== Week 2 - Feb 1: The evolution of Data Management and introduction to Big Data; Introduction to Databases, Relational Model and SQL==

* '''Lecture notes:'''
** http://vgc.poly.edu/~juliana/courses/BigData2016/Lectures/datamanagement.pdf
** http://vgc.poly.edu/~juliana/courses/BigData2016/Lectures/intro-db.pdf
* '''Lab:''' in-class assignment on relational algebra
* '''Readings:'''
** [http://philip.greenspun.com/sql/introduction.html Greenspun's SQL for Web Nerds Intro]
** [http://philip.greenspun.com/sql/data-modeling.html SQL/Nerds Modeling (parts)]

== Week 3 - Feb 8: Introduction to Databases, Relational Model and SQL (cont.) ==

*''' Lecture notes:''' http://vgc.poly.edu/~juliana/courses/BigData2016/Lectures/intro-db.pdf
* '''Lab:''' SQL
* '''Programming assignment:''' Using SQL for data analysis and cleaning (check NYU Classes)

== Week 4 - Feb 15: Holiday ==

= Big Data Foundations and Infrastructure (3 weeks) =

== Week 5 - Feb 22: Introduction to Map Reduce ==

*''' Lecture notes:''' http://vgc.poly.edu/~juliana/courses/BigData2016/Lectures/mapreduce-intro.pdf
* '''Lab:''' Hands-on Hadoop (local and AWS)
* Quiz 1 (Map Reduce) -- check http://www.newgradiance.com/services

== Week 6 - Feb 29: MapReduce Algorithm Design Patterns ==

*''' Lecture notes:''' http://vgc.poly.edu/~juliana/courses/BigData2015/Lectures/mapreduce-algo-design.pdf
* '''Lab:''' Hands-on Hadoop (HPC)
* '''Programming assignment:''' Map Reduce (check NYU Classes)
* '''Readings''':
** Data-Intensive Text Processing with MapReduce, Chapters 1 and 2
** Data-Intensive Text Processing with MapReduce (Jan 27, 2013), Chapter 6 -- Processing Relational Data (this chapter appears in the 2013 version of the textbook -- http://lintool.github.io/MapReduceAlgorithms/ed1n/MapReduce-algorithms.pdf)

== Week 7 - March 7: Parallel Databases vs MapReduce; Storage Solutions; Introduction to SPARK==

*''' Lecture notes:''' ** http://vgc.poly.edu/~juliana/courses/BigData2016/Lectures/paralleldb-vs-hadoop.pdf
* '''Lab:''' NoSQL
* '''Programming assignment:''' Pig and Spark
* '''Readings''':
** Benchmark DBMS vs MapReduce (2009): http://database.cs.brown.edu/sigmod09/benchmarks-sigmod09.pdf
** MapReduce: A Flexible Data Processing Tool: http://cacm.acm.org/magazines/2010/1/55744-mapreduce-a-flexible-data-processing-tool/fulltext

* '''Additional Suggested reading:'''
** Hive - A Warehousing Solution Over a Map-Reduce Framework. http://dl.acm.org/citation.cfm?id=1687609
** Pig Latin: A Not-So-Foreign Language for Data Processing. http://dl.acm.org/citation.cfm?id=1376726
** BigTable: http://fcoffice.googlecode.com/svn/%E4%B9%A6%E7%B1%8D/bigtable-osdi06.pdf
** Spark: Cluster Computing with Working Sets. http://static.usenix.org/legacy/events/hotcloud10/tech/full_papers/Zaharia.pdf

== Week 8 -- March 14th: Spring Break ==

= Transparency and Reproducibility (1 week) =

== Week 9 - March 21: Data Exploration and Reproducibility ==

* Lecture notes: http://vgc.poly.edu/~juliana/courses/BigData2015/Lectures/data-science-reproducibility.pdf

* Lab: Hands-on reproducibility. Before class, please
** Download VisTrails 2.1.5 from here: http://www.vistrails.org/index.php/Downloads
** Download the mta-analysis example: http://bigdata.poly.edu/~fchirigati/mda-class/mta-analysis.vt
** Download the links for the input data: http://bigdata.poly.edu/~fchirigati/mda-class/mta-links.txt
** http://bigdata.poly.edu/~fchirigati/mda-class/hands-on.pdf
** Questions? Email Fernando at fchirigati@nyu.edu

* Programming assignment 4: Exploring urban data (see NYU Classes)

= Big Data Algorithms, Mining Techniques, and Visualization (6 weeks) =

== Week 10 - March 28th: Finding similar items ==

* Lecture notes:
** http://vgc.poly.edu/~juliana/courses/BigData2015/Lectures/similarity.pdf

* Reading: Chapter 3 [http://vgc.poly.edu/~juliana/courses/BigData2015/Textbooks/ullman-book-v1.1-mining-massive-data.pdf Mining of Massive Datasets]

* Homework Assignment
** See quizzes on [http://www.newgradiance.com Gradiance] -- Distance measures and document similarity.

== Week 11 - April 4th: Association Rules ==

* Lecture notes:
** http://vgc.poly.edu/~juliana/courses/BigData2015/Lectures/association-rules.pdf

* Reading: Chapter 6 [http://vgc.poly.edu/~juliana/courses/MassiveDataAnalysis2014/Textbooks/ullman-book-v1.1-mining-massive-data.pdf Mining of Massive Datasets]

* Suggested additional reading:
**Fast algorithms for mining association rules, Agrawal and Srikant, VLDB 1994.
**Data Mining Concepts and Techniques, Jiawei Han and Micheline Kamber, Morgan Kaufmann
**Dynamic Itemset Counting and Implication Rules for Market Basket Data. Brin et al., SIGMOD 1997. http://www-db.stanford.edu/~sergey/dic.html

* Homework Assignment
** See quizes on [http://www.newgradiance.com Gradiance] -- Distance measures and document similarity.

== Week 12 - April 11th: Visualization and Spatio-Temporal Data -- Invited lecture by Dr. Harish Doraiswamy (NYU CUSP) ==

* Lecture notes:
** http://vgc.poly.edu/~juliana/courses/BigData2015/Lectures/SpatialQP.pdf

* Lab: Using Amazon AWS to analyze and visualize taxi data
** https://github.com/ViDA-NYU/aws_taxi

== Week 13 - April 18th: Parallel Databases ==

* Lecture notes:
** http://vgc.poly.edu/~juliana/courses/BigData2015/Lectures/paralleldb-vs-hadoop-2015.pdf

* Required reading:
** Benchmark DBMS vs MapReduce (2009): http://database.cs.brown.edu/sigmod09/benchmarks-sigmod09.pdf
** MapReduce: A Flexible Data Processing Tool: http://cacm.acm.org/magazines/2010/1/55744-mapreduce-a-flexible-data-processing-tool/fulltext

* Suggested reading:
** Hive - A Warehousing Solution Over a Map-Reduce Framework. http://dl.acm.org/citation.cfm?id=1687609
** Pig Latin: A Not-So-Foreign Language for Data Processing. http://dl.acm.org/citation.cfm?id=1376726
** BigTable: http://fcoffice.googlecode.com/svn/%E4%B9%A6%E7%B1%8D/bigtable-osdi06.pdf
** Spark: Cluster Computing with Working Sets. http://static.usenix.org/legacy/events/hotcloud10/tech/full_papers/Zaharia.pdf

== Week 14 - April 25th: Graph Analysis ==

* Lecture notes:
** http://vgc.poly.edu/~juliana/courses/BigData2015/Lectures/graph-algos.pdf

* Required Reading: Data-Intensive Text Processing with MapReduce. Chapters 5 -- Graph Algorithms

== Week 15 - May 2: Final Exam ==

== Week 16 - May 9: Project Presentations ==

== Week 17 - May 16: Project Presentations ==

Course: Big Data 2016

2016-01-23T21:55:36Z

Juliana:

Course: Big Data 2016

2016-01-23T21:47:14Z

Juliana: /* News */

Course: Big Data 2016

2016-01-23T21:46:02Z

Juliana:

2016-01-06T22:27:53Z

Juliana: Created page with '= DS-GA 1004- Big Data: Tentative Schedule -- ''subject to change'' = * Course Web page: http://vgc.poly.edu/~juliana/courses/BigData2016 * Instructor: Professor Juliana Freire…'

= DS-GA 1004- Big Data: Tentative Schedule -- ''subject to change'' =

* Course Web page: http://vgc.poly.edu/~juliana/courses/BigData2016

* Instructor: Professor Juliana Freire (http://vgc.poly.edu/~juliana)

* Lecture: Mondays, 4:55pm-7:35pm at 19 University Pl., room 102.
* Some classes will include a lab session, please "always bring your laptop.''

= News =

* 1/25/2016: Amazon has kindly donated time on AWS for all the student in this class. To obtain your credit, please follow the instructions at http://www.vistrails.org/index.php/AWS_Setup
* 1/25/2016: Access you NYU HPC account, which you will use for in-class exercises and homework assignments. See instructions at [[NYU Hadoop]]

= Background (2 weeks) =

== Week 1 - Feb 2: Course Overview; The evolution of Data Management and introduction to Big Data ==

* Lecture notes: http://vgc.poly.edu/~juliana/courses/BigData2015/Lectures/course-overview.pdf
* Reading: Chapter 1 of Mining of Massive Data Sets (version 1.1)
* Course survey: https://docs.google.com/forms/d/1LTiJwkDVvp0cF62Fw_d9Y86US5LCkorRUIQtV2T8KWE/viewform?usp=send_form

== Week 2 - Feb 9: Introduction to Databases, Relational Model and SQL ==
* Lecture notes:
** http://vgc.poly.edu/~juliana/courses/BigData2015/Lectures/intro-to-db.pdf
** http://vgc.poly.edu/~juliana/courses/BigData2015/Lectures/relational-algebra.pdf
** http://vgc.poly.edu/~juliana/courses/BigData2015/Lectures/sql-intro.pdf
** http://vgc.poly.edu/~juliana/courses/BigData2015/Lectures/sql-more.pdf

* Lab:
** SQL hands on: [[Big Data 2015 - SQL Lab]]

* Other useful reading:
** [http://philip.greenspun.com/sql/introduction.html Greenspun's SQL for Web Nerds Intro]
** [http://philip.greenspun.com/sql/data-modeling.html SQL/Nerds Modeling (parts)]

* Programming assignment: Using SQL for data analysis and cleaning (see NYU Classes)

== Feb 16: Holiday ==

= Big Data Foundations and Infrastructure (3 weeks) =

== Week 3 - Feb 23: Introduction to Map Reduce ==
* Lab: (continuation)
** SQL hands on: [[Big Data 2015 - SQL Lab]]
* Lecture notes:
** http://vgc.poly.edu/~juliana/courses/BigData2015/Lectures/mapreduce-intro.pdf
* Required Reading:
** Data-Intensive Text Processing with MapReduce. Chapters 1 and 2
** Mining of Massive Datasets (v 2.1). Chapter 2 - 2.1, 2.2, and 2.3
* Other useful reading:
** Hadoop: The Definitive Guide. http://www.amazon.com/Hadoop-Definitive-Guide-Tom-White/dp/1449311520

* Quiz 1 (Map Reduce) assigned -- check http://www.newgradiance.com/services

== Week 4 - March 2: Algorithm Design for MapReduce: Relational Operations ==

* Lecture notes:
** http://vgc.poly.edu/~juliana/courses/BigData2015/Lectures/mapreduce-algo-design-relations.pdf

* Lab: Hands-on Hadoop (local)

* Required reading:
** Data-Intensive Text Processing with MapReduce, Chapters 1 and 2
** Mining of Massive Datasets (2nd Edition), Chapter 2.

* Programming assignment: Map Reduce (check NYU Classes)

== Week 5 - March 9: MapReduce Algorithm Design Patterns; Parallel Databases vs MapReduce ==

* Lecture notes:
** http://vgc.poly.edu/~juliana/courses/BigData2015/Lectures/mapreduce-algo-design-patterns.pdf

* Lab: Hands-on Hadoop on AWS
** Lab materials: http://bigdata.poly.edu/~tuananh/files/awscli-examples.zip
** Install aws command-line interface: http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/set-up-ec2-cli-linux.html

* Some links to AWS CLI documentation:
** http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/set-up-ec2-cli-linux.html
** http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-set-up.html
** http://www.linux.com/learn/tutorials/761430-an-introduction-to-the-aws-command-line-tool
**EMR Through Commandline: https://www.safaribooksonline.com/library/view/programming-elastic-mapreduce/9781449364038/ch04.html
** Importing Key: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html#how-to-generate-your-own-key-and-import-it-to-aws
** EMR Job Flow: http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/EMR_CreateJobFlow.html

* Required reading:
** Data-Intensive Text Processing with MapReduce, Chapters 1 and 2
** Data-Intensive Text Processing with MapReduce (Jan 27, 2013), Chapter 6 -- Processing Relational Data (this chapter appears in the 2013 version of the textbook -- http://lintool.github.io/MapReduceAlgorithms/ed1n/MapReduce-algorithms.pdf)

* Programming assignment: check NYU Classes on March 10th

== March 16th: Spring Break ==

= Transparency and Reproducibility (1 week) =

== Week 6 - March 23: Data Exploration and Reproducibility ==

* Lecture notes: http://vgc.poly.edu/~juliana/courses/BigData2015/Lectures/data-science-reproducibility.pdf

* Lab: Hands-on reproducibility. Before class, please
** Download VisTrails 2.1.5 from here: http://www.vistrails.org/index.php/Downloads
** Download the mta-analysis example: http://bigdata.poly.edu/~fchirigati/mda-class/mta-analysis.vt
** Download the links for the input data: http://bigdata.poly.edu/~fchirigati/mda-class/mta-links.txt
** http://bigdata.poly.edu/~fchirigati/mda-class/hands-on.pdf
** Questions? Email Fernando at fchirigati@nyu.edu

* Programming assignment 4: Exploring urban data (see NYU Classes)

= Big Data Algorithms, Mining Techniques, and Visualization (6 weeks) =

== Week 7 - March 30th: Finding similar items ==

* Lecture notes:
** http://vgc.poly.edu/~juliana/courses/BigData2015/Lectures/similarity.pdf

* Reading: Chapter 3 [http://vgc.poly.edu/~juliana/courses/BigData2015/Textbooks/ullman-book-v1.1-mining-massive-data.pdf Mining of Massive Datasets]

* Homework Assignment
** See quizzes on [http://www.newgradiance.com Gradiance] -- Distance measures and document similarity.

== Week 8 - April 6th: Association Rules ==

* Lecture notes:
** http://vgc.poly.edu/~juliana/courses/BigData2015/Lectures/association-rules.pdf

* Reading: Chapter 6 [http://vgc.poly.edu/~juliana/courses/MassiveDataAnalysis2014/Textbooks/ullman-book-v1.1-mining-massive-data.pdf Mining of Massive Datasets]

* Suggested additional reading:
**Fast algorithms for mining association rules, Agrawal and Srikant, VLDB 1994.
**Data Mining Concepts and Techniques, Jiawei Han and Micheline Kamber, Morgan Kaufmann
**Dynamic Itemset Counting and Implication Rules for Market Basket Data. Brin et al., SIGMOD 1997. http://www-db.stanford.edu/~sergey/dic.html

* Homework Assignment
** See quizes on [http://www.newgradiance.com Gradiance] -- Distance measures and document similarity.

== Week 9 - April 13th: Visualization and Spatio-Temporal Data -- Invited lecture by Dr. Harish Doraiswamy (NYU CUSP) ==

* Lecture notes:
** http://vgc.poly.edu/~juliana/courses/BigData2015/Lectures/SpatialQP.pdf

* Lab: Using Amazon AWS to analyze and visualize taxi data
** https://github.com/ViDA-NYU/aws_taxi

== Week 10 - April 20th: Parallel Databases ==

* Lecture notes:
** http://vgc.poly.edu/~juliana/courses/BigData2015/Lectures/paralleldb-vs-hadoop-2015.pdf

* Required reading:
** Benchmark DBMS vs MapReduce (2009): http://database.cs.brown.edu/sigmod09/benchmarks-sigmod09.pdf
** MapReduce: A Flexible Data Processing Tool: http://cacm.acm.org/magazines/2010/1/55744-mapreduce-a-flexible-data-processing-tool/fulltext

* Suggested reading:
** Hive - A Warehousing Solution Over a Map-Reduce Framework. http://dl.acm.org/citation.cfm?id=1687609
** Pig Latin: A Not-So-Foreign Language for Data Processing. http://dl.acm.org/citation.cfm?id=1376726
** BigTable: http://fcoffice.googlecode.com/svn/%E4%B9%A6%E7%B1%8D/bigtable-osdi06.pdf
** Spark: Cluster Computing with Working Sets. http://static.usenix.org/legacy/events/hotcloud10/tech/full_papers/Zaharia.pdf

== Week 11 - April 27th: Graph Analysis ==

* Lecture notes:
** http://vgc.poly.edu/~juliana/courses/BigData2015/Lectures/graph-algos.pdf

* Required Reading: Data-Intensive Text Processing with MapReduce. Chapters 5 -- Graph Algorithms

== Week 12 - May 4: Final Exam ==

== Week 13 - May 11: Project Presentations ==

== Week 14 - May 18: Project Presentations ==

AWS Setup

2016-01-06T22:14:31Z

Juliana: /* Setting up your AWS account */

==Setting up your AWS account==

* First you need to create an AWS account (if you do not have one already).
Go to http://aws.amazon.com/ and sign up: You can create a new account by selecting "I am a new user."
Enter your contact information and confirm your acceptance of the AWS Customer Agreement.
Once you have created an Amazon Web Services Account, check your email for your confirmation step. You need Access Identifiers to make valid web service requests.

You need to sign up for three of their services: Simple Storage Service (S3), Elastic Compute Cloud (EC2), and Amazon Elastic MapReduce. You can do this by clicking on "Sign in to the AWS Management Console".

* Then, apply to the AWSEducate program. Follow the instructions at https://www.awseducate.com/Application
As part of this process, you will need to provide your AWS account number.

* After your application is reviewed and approved, you will receive a welcome email including details about your AWS credits. Go to http://aws.amazon.com/awscredits and redeem your credit.

"Very important: Be aware that '''if you exceed it, Amazon will charge your credit card without warning'''. This credit should be enough for your assignments (if you are interested in their changes, see AWS charges: currently, AWS charges about 8 cents/node/hour for the default "small" node size.). You must remember to terminate manually the AWS cluster (called Job Flows) when you are done: if you just close the browser, the job flows continue to run, and amazon will continue to charge you for days and weeks, exhausting your credit and charging you huge amount on your credit card.''' Remember to terminate the AWS cluster."

== Setting up an EC2 key pair ==

See http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/generating-a-keypair.html#how-to-have-aws-create-the-key-pair-for-you

To connect to an Amazon EC2 node, such as the master nodes for the Hadoop clusters you will be creating, you need an SSH key pair. To create and install one, do the following:

After setting up your account, follow Amazon's instructions to create a key pair. Follow the instructions in section "Having AWS create the key pair for you," subsection "AWS Management Console." (Don't do this in Internet Explorer, or you might not be able to download the .pem private key file.)

Download and save the .pem private key file to disk. We will reference the .pem file as </path/to/saved/keypair/file.pem> in the following instructions.

Make sure only you can access the .pem file, just to be safe:

<code>
$ chmod 600 </path/to/saved/keypair/file.pem>
</code>

== Terminating an AWS cluster ==

After you are done, shut down the AWS cluster:
<code>
Go to the Management Console.
Select the job in the list.
Click the Terminate button (it should be right below "Your Elastic MapReduce Job Flows").
Wait for a while (may take minutes) and recheck until the job state becomes TERMINATED.
</code>

Pay attention to this step. If you fail to terminate your job and only close the browser, or log off AWS, your AWS will continue to run, and AWS will continue to charge you: for hours, days, weeks, and when your credit is exhausted, it charges your credit card. Make sure you don't leave the console until you have confirmation that the job is terminated.

== Monitoring Hadoop Jobs ==

You are required in this assignment to monitor the running Hadoop jobs on your AWS cluster using the master node's job tracker Web UI. There are two ways to do this: using lynx or using your own browser with a SOCKS proxy.

=== Using LYNX ===

Very easy, you don't need to download anything. Open a separate ssh connection to the AWS master node and type:

<code>
% lynx http://localhost:9100/
</code>

Lynx is a text browser. Navigate as follows: up/down arrows = move through the links (current link is highlighted); enter = follows a link; left arrow = return to previous page.

Examine the webpage carefully, while your program is running. You should find information about the map tasks, the reduce tasks, you should be able to drill down into each map task (for example to monitor its progress); you should be able to look at the log files of the map tasks (if there are runtime errors, you will see them only in these log files).

=== Using SOCKS proxy, and your own browser ===

This requires more work, but the nicer interface makes it worth the extra work. Set up your browser to use a proxy when connecting to the master node.

* Firefox:
<code>
Install the FoxyProxy extension for Firefox.
Copy the foxyproxy.xml configuration file from the hw6/ folder into your Firefox profile folder.
If the previous step doesn't work for you, try deleting the foxyproxy.xml you copied into your profile, and using Amazon's instructions to set up FoxyProxy manually.
</code>

* Chrome:
<code>
Install proxy switch!, by clicking "Add to Chrome" on the extension's page.
After clicking the link, you should be at the ProxySwitch options page, but if not, click the Tools wrench icon (upper right corner). Go to Options, go to Extensions. Here you will see the ProxySwitch!: click on Options next to it.
Create a new Proxy Profile: Manual Configuration, Profile name = Amazon Elastic MapReduce (any name you want), SOCKS Host = localhost, Port = 8888 (you can choose any port you want; another favorite is 8157), SOCKS v5.
Create two new swtich rules (give them any names, say AWS1 and AWS2). Rule 1: pattern=*.amazonaws.com:*/*, Rule 2: pattern=*.ec2.internal:*/*. For both, Type=wildcard, Proxy profile=[the profile you created at the previous step].
Open a new local terminal window and create the SSH SOCKS tunnel to the master node using the following:

$ ssh -o "ServerAliveInterval 10" -i </path/to/saved/keypair/file.pem> -ND 8888 hadoop@<master.public-dns-name.amazonaws.com>

(The -N option tells ssh not to start a shell, and the -D 8888 option tells ssh to start the proxy and have it listen on port 8888.)
</code>

The resulting SSH window will appear to hang, without any output; this is normal as SSH has not started a shell on the master node, but just created the tunnel over which proxied traffic will run.

Keep this window running in the background (minimize it) until you are finished with the proxy, then close the window to shut the proxy down. Open your browser, and type one of the following URLs:
<code>
For the job tracker: http://<master.public-dns-name.amazonaws.com>:9100/
For HDFS management: http://<master.public-dns-name.amazonaws.com>:9101/
</code>

== Killing a Hadoop Job ==

From the job tracker interface find the hadoop job_id, then type:

% hadoop job -kill job_id

== Managing the results of your tasks ==

=== Copying files to or from the AWS master node ===

To copy one file from the master node back to your computer, run this command on the local computer:

<code>
$ scp -o "ServerAliveInterval 10" -i </path/to/saved/keypair/file.pem> hadoop@<master.public-dns-name.amazonaws.com>:<file_path> .
</code>

where <file_path> can be absolute or relative to the AWS master node's home folder. The file should be copied onto your current directory ('.') on your local computer.

Better: copy an entire directory, recursively. Suppose your files are in the directory example-results. They type the following on your loal computer:

<code>
$ scp -o "ServerAliveInterval 10" -i </path/to/saved/keypair/file.pem> -r hadoop@<master.public-dns-name.amazonaws.com>:example-results .
</code>

As an alternative, you may run the scp command on the AWS master node, and connect to your local machine. For that, you need to know your local machine's domain name, or IP address, and your local machine needs to accept ssh connections.

=== Storing Files in S3 ===

This seems much easier to use. Go to your AWS Management Console, click on Create Bucket, and create a new bucket (=directory). Give it a name that may be a public name. Let's say you call it superman-hw6. Click on the Properties button, then Permissions tab. Make sure you have all the permissions.

In your program, you can write the results to 's3n://superman-hw6/example-results'. When the program terminates, then in your S3 console you should see the new directory example-results. Click on individual files to download. The number of files depends on the number of reduce tasks, and may vary from one to a few dozens. The only disadvantage of using S3 is that you have to click on each file separately to download.

Note that S3 is permanent storage, and you are charged for it. You can safely store all your query answers for several weeks without exceeding your credit; at some point in the future remember to delete them.

''Modified from http://www.cs.washington.edu/education/courses/csep544/11au/hw/hw6/hw6-awsusage.html''