Course: Big Data Analysis

Make sure to check my.poly.edu for course announcements

Week 1: Monday Sept. 10th - Course Overview

• Course overview (First day of classes!)
• Student survey
• Introduction to Big Data

Readings

• Dilbert's BigData
• New York Time's "How BigData Became so Big"
• World Economic Forum: Big Data, Big Impact
• The Analytics Journey
• BigData Analytics Usecases
• Data-Intensive Text Processing with MapReduce, Chapter1
• PDMBS vs. MapReduce
• Benchmark DBMS vs MapReduce (2009)

Week 2: Monday Sept. 17th - Map-Reduce

• Introduction to map-reduce
• Introduction to Hadoop
• Map-Reduce ecosystem: Pig, Hive, Jaql, Mahout, BigInsights

Readings

• original google map-reduce paper
• Mining of Massive Datasets, Chapter 2
• Data-Intensive Text Processing with MapReduce, Chapter 2, Chapter 3
• Pig Latin: A Not-So-Foreign Language for Data Processing
• Jaql: A Scripting Language for Large Scale Semistructured Data Analysis
• Hive - A Warehousing Solution Over a Map-Reduce Framework

Week 3: Monday Sept. 24th - Statistics is easy - Invited Speaker: Dennis Shasha

• Guest lecture by Dennis Shasha
• Statistics and Big Data

Readings

• http://www.morganclaypool.com/doi/abs/10.2200/S00142ED1V01Y200807MAS001 -- book is available for free for NYU students
• JF: add references for issues related to stats and big data

Week 4: Monday Oct. 1st - Databases and Big Data

• Databases and Big Data: Persistence, Querying, Indexing, Transactions
• BigTables and NoSQL stores. Tuple store vs. column stores: HBase, MongoDB, Cassandra
• "NewSQL" stores: more on Hive, VoltDB, HadoopDB,
• Transactions in NoSQL stores
• Beyond MapReduce: Berkeley's Spark, UC Irvine's Asterix, Google's Dremel

Readings

• HadoopDB: An Architectural Hybrid of MapReduce and DBMS Technologies for Analytical Workloads
• Low Overhead Concurrency Control for Partitioned Main Memory Databases
• ASTERIX: Towards a Scalable, Semistructured Data Platform for Evolving-World Models.
• Dremel: Interactive Analysis of Web-Scale Datasets
• Large-scale Incremental Processing Using Distributed Transactions and Notifications

Week 5: Monday Oct. 8st - Finding Similar Items

• Overview of information integration

Readings

• Mining of Massive Datasets, chapter 3; information integration; entity resolution

Week 6: Monday Oct. 15st - Invited Speaker: Torsten Suel

• Reading: inverted index and crawling (Lin chapter 4)
• Ask Torsten (tentative, ask him for reading material)

Readings

• Mining of Massive Datasets, Chapter 5
• Data-Intensive Text Processing with MapReduce, Chapter 5

Week 7: Monday Oct. 22st - Invited Speakers: Claudio Silva and Lauro Lins

• Introduction to Visualization; Data stewardship and provenance
• Guest lecture by Claudio Silva and Lauro Lins

Readings

• Hellerstein (ask Claudio for additional references)
• ADD: provenance and reproducibility

Week 8: Monday Oct. 29th - Graph Analysis

• Graph algorithms, link analysis, social networks

Readings

• Data-Intensive Text Processing with MapReduce, Chapter 4

Week 9: Monday Nov. 5th - Frequent Itemsets

Reading

• Mining of Massive Datasets, Chapter 6

Week 10: Monday Nov. 12th - Mining Data Streams =

Readings

• Mining of Massive Datasets, Chapter 4

Week 11: Monday Nov. 19th - Clustering

Readings

• Mining of Massive Datasets, Chapter 7

Week 12: Monday Nov. 26th - Recommendation Systems

Readings

• Mining of Massive Datasets, Chapter 9

Week 13 Monday Dec. 3rd - EM algorithms for text processing

• Data-Intensive Text Processing with MapReduce, Chapter 6

Further Readings

• Anomaly Detection: A Survey