Course: Big Data 2015
Revision as of 23:12, 23 March 2015 by Fchirigati (talk | contribs) (→Week 6 - March 23: Data Exploration and Reproducibility)
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.
DS-GA 1004- Big Data: Tentative Schedule -- subject to change
- Course Web page: http://vgc.poly.edu/~juliana/courses/BigData2015
- Instructor: Professor Juliana Freire (http://vgc.poly.edu/~juliana)
- Lecture: Mondays, 4:55pm-7:35pm at Silver, room 208.
- Some classes will include a lab session, please "always bring your laptop.
News
- Big Data 2015: Final Project
- 2/26/2015: An Amazon AWS token was emailed to each student. Please create your Amazon AWS account. You can find instructions at: http://www.vistrails.org/index.php/AWS_Setup
- 2/26/2015: You should install the Cloudera VM on your laptop. We will need that for the lab on March 9th. Here are the instructions: Cloudera VM Setup
- There is a new version of the textbook Mining of Massive Datasets, we will use the latest version 2.1
Background (2 weeks)
Week 1 - Feb 2: Course Overview; The evolution of Data Management and introduction to Big Data
- Lecture notes: http://vgc.poly.edu/~juliana/courses/BigData2015/Lectures/course-overview.pdf
- Reading: Chapter 1 of Mining of Massive Data Sets (version 1.1)
- Course survey: https://docs.google.com/forms/d/1LTiJwkDVvp0cF62Fw_d9Y86US5LCkorRUIQtV2T8KWE/viewform?usp=send_form
Week 2 - Feb 9: Introduction to Databases, Relational Model and SQL
- Lecture notes:
- Lab:
- SQL hands on: Big Data 2015 - SQL Lab
- Other useful reading:
- Programming assignment: Using SQL for data analysis and cleaning (see NYU Classes)
Feb 16: Holiday
Big Data Foundations and Infrastructure (3 weeks)
Week 3 - Feb 23: Introduction to Map Reduce
- Lab: (continuation)
- SQL hands on: Big Data 2015 - SQL Lab
- Lecture notes:
- Required Reading:
- Data-Intensive Text Processing with MapReduce. Chapters 1 and 2
- Mining of Massive Datasets (v 2.1). Chapter 2 - 2.1, 2.2, and 2.3
- Other useful reading:
- Hadoop: The Definitive Guide. http://www.amazon.com/Hadoop-Definitive-Guide-Tom-White/dp/1449311520
- Quiz 1 (Map Reduce) assigned -- check http://www.newgradiance.com/services
Week 4 - March 2: Algorithm Design for MapReduce: Relational Operations
- Lecture notes:
- Lab: Hands-on Hadoop (local)
- Required reading:
- Data-Intensive Text Processing with MapReduce, Chapters 1 and 2
- Mining of Massive Datasets (2nd Edition), Chapter 2.
- Programming assignment: Map Reduce (check NYU Classes)
Week 5 - March 9: MapReduce Algorithm Design Patterns; Parallel Databases vs MapReduce
- Lecture notes:
- Lab: Hands-on Hadoop on AWS
- Lab materials: http://bigdata.poly.edu/~tuananh/files/awscli-examples.zip
- Install aws command-line interface: http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/set-up-ec2-cli-linux.html
- Some links to AWS CLI documentation:
- http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/set-up-ec2-cli-linux.html
- http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-set-up.html
- http://www.linux.com/learn/tutorials/761430-an-introduction-to-the-aws-command-line-tool
- EMR Through Commandline: https://www.safaribooksonline.com/library/view/programming-elastic-mapreduce/9781449364038/ch04.html
- Importing Key: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html#how-to-generate-your-own-key-and-import-it-to-aws
- EMR Job Flow: http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/EMR_CreateJobFlow.html
- Required reading:
- Data-Intensive Text Processing with MapReduce, Chapters 1 and 2
- Data-Intensive Text Processing with MapReduce (Jan 27, 2013), Chapter 6 -- Processing Relational Data (this chapter appears in the 2013 version of the textbook -- http://lintool.github.io/MapReduceAlgorithms/ed1n/MapReduce-algorithms.pdf)
- Programming assignment: check NYU Classes on March 10th
March 16th: Spring Break
Transparency and Reproducibility (1 week)
Week 6 - March 23: Data Exploration and Reproducibility
- Lecture notes: http://vgc.poly.edu/~juliana/courses/BigData2015/Lectures/data-science-reproducibility.pdf
- Lab: Hands-on reproducibility. Before class, please
- Download VisTrails 2.1.5 from here: http://www.vistrails.org/index.php/Downloads
- Download the mta-analysis example: http://bigdata.poly.edu/~fchirigati/mda-class/mta-analysis.vt
- Download the links for the input data: http://bigdata.poly.edu/~fchirigati/mda-class/mta-links.txt
- http://bigdata.poly.edu/~fchirigati/mda-class/hands-on.pdf
- Programming assignment 4: Exploring urban data (see NYU Classes)
Big Data Algorithms, Mining Techniques, and Visualization (6 weeks)
Week 7 - March 30th: Finding similar items
- Reading: Chapter 3 Mining of Massive Datasets
- Homework Assignment
- See quizes on Gradiance -- Distance measures and document similarity.
Week 8 - April 6th: Association Rules
- Lecture notes:
- Assignment on frequent items and association rule mining. Due on Dec 7th. Check http://www.newgradiance.com/services
- Reading: Chapter 6 Mining of Massive Datasets
- Suggested additional reading:
- Fast algorithms for mining association rules, Agrawal and Srikant, VLDB 1994.
- Data Mining Concepts and Techniques, Jiawei Han and Micheline Kamber, Morgan Kaufmann
- Dynamic Itemset Counting and Implication Rules for Market Basket Data. Brin et al., SIGMOD 1997. http://www-db.stanford.edu/~sergey/dic.html
Week 9 - April 13th: Visualization and Spatio-Temporal Data -- Invited lecture by Dr. Huy Vo (NYU CUSP)
- Lecture notes:
Week 10: Parallel Databases
- Benchmark DBMS vs MapReduce (2009): http://database.cs.brown.edu/sigmod09/benchmarks-sigmod09.pdf
- MapReduce: A Flexible Data Processing Tool: http://cacm.acm.org/magazines/2010/1/55744-mapreduce-a-flexible-data-processing-tool/fulltext