Course: Big Data 2016 - Revision history

Juliana at 04:23, 30 January 2017

2017-01-30T04:23:24Z

← Older revision		Revision as of 04:23, 30 January 2017
Line 1:		Line 1:
	= DS-GA 1004- Big Data: Tentative Schedule -- ''subject to change'' =		= DS-GA 1004- Big Data: Tentative Schedule -- ''subject to change'' =

			[[Course: Big Data 2017]]

	* Course Web page: http://vgc.poly.edu/~juliana/courses/BigData2016		* Course Web page: http://vgc.poly.edu/~juliana/courses/BigData2016

Juliana at 19:41, 25 April 2016

2016-04-25T19:41:53Z

← Older revision		Revision as of 19:41, 25 April 2016
Line 147:		Line 147:
	** http://vgc.poly.edu/~juliana/courses/BigData2016/Lectures/SpatialQP-2016.pdf		** http://vgc.poly.edu/~juliana/courses/BigData2016/Lectures/SpatialQP-2016.pdf

			* Lab: see NYU Classes

Juliana at 19:41, 25 April 2016

2016-04-25T19:41:23Z

← Older revision		Revision as of 19:41, 25 April 2016
Line 140:		Line 140:
	* Bio: Divesh Srivastava is the head of Database Research at AT&T Labs-Research. He is a Fellow of the Association for Computing Machinery (ACM) and the managing editor of the Proceedings of the VLDB Endowment (PVLDB). He received his Ph.D. from the University of Wisconsin, Madison, USA, and his Bachelor of Technology from the Indian Institute of Technology, Bombay, India. His research interests and publications span a variety of topics in data management.		* Bio: Divesh Srivastava is the head of Database Research at AT&T Labs-Research. He is a Fellow of the Association for Computing Machinery (ACM) and the managing editor of the Proceedings of the VLDB Endowment (PVLDB). He received his Ph.D. from the University of Wisconsin, Madison, USA, and his Bachelor of Technology from the Indian Institute of Technology, Bombay, India. His research interests and publications span a variety of topics in data management.

	* Lecture notes:		* Lecture notes: http://vgc.poly.edu/~juliana/courses/BigData2016/Lectures/bdq.pdf

	== Week 14 - April 25th: Exploring Spatio-Temporal Data -- Invited lecture by Dr. Harish Doraiswamy (NYU CDS) ==		== Week 14 - April 25th: Exploring Spatio-Temporal Data -- Invited lecture by Dr. Harish Doraiswamy (NYU CDS) ==

	* Lecture notes:		* Lecture notes:
	** http://vgc.poly.edu/~juliana/courses/BigData2016/Lectures/SpatialQP.pdf		** http://vgc.poly.edu/~juliana/courses/BigData2016/Lectures/SpatialQP-2016.pdf


	* Lab: Using Amazon AWS to analyze and visualize taxi data
	** https://github.com/ViDA-NYU/aws_taxi

Juliana: /* Week 13 - April 18th: Data quality: the other face of big data - Invited lecture by Dr. Divesh Srivastava, AT&T Research */

2016-04-25T19:40:26Z

Week 13 - April 18th: Data quality: the other face of big data - Invited lecture by Dr. Divesh Srivastava, AT&T Research

← Older revision		Revision as of 19:40, 25 April 2016
Line 138:		Line 138:
	* Abstract: In our Big Data era, data is being generated, collected and analyzed at an unprecedented scale, and data-driven decision making is sweeping through all aspects of society. Recent studies have shown that poor quality data is prevalent in large databases and on the Web. Since poor quality data can have serious consequences on the results of data analyses, the importance of veracity, the fourth “V” of big data is increasingly being recognized. In this talk, we highlight the substantial challenges that the first three “V”s, volume, velocity and variety, bring to dealing with veracity in big data. Due to the sheer volume and velocity of data, one needs to understand and (possibly) repair erroneous data in a scalable and timely manner. With the variety of data, often from a diversity of sources, data quality rules cannot be specified a priori; one needs to let the “data to speak for itself” in order to discover the semantics of the data. This talk presents recent results that are relevant to big data quality management, focusing on the two major dimensions of (i) discovering quality issues from the data itself, and (ii) trading-off accuracy vs efficiency.		* Abstract: In our Big Data era, data is being generated, collected and analyzed at an unprecedented scale, and data-driven decision making is sweeping through all aspects of society. Recent studies have shown that poor quality data is prevalent in large databases and on the Web. Since poor quality data can have serious consequences on the results of data analyses, the importance of veracity, the fourth “V” of big data is increasingly being recognized. In this talk, we highlight the substantial challenges that the first three “V”s, volume, velocity and variety, bring to dealing with veracity in big data. Due to the sheer volume and velocity of data, one needs to understand and (possibly) repair erroneous data in a scalable and timely manner. With the variety of data, often from a diversity of sources, data quality rules cannot be specified a priori; one needs to let the “data to speak for itself” in order to discover the semantics of the data. This talk presents recent results that are relevant to big data quality management, focusing on the two major dimensions of (i) discovering quality issues from the data itself, and (ii) trading-off accuracy vs efficiency.

	* Bio: Divesh Srivastava is the head of Database Research at AT&T Labs-Research. He is a Fellow of the Association for Computing Machinery (ACM) and the managing editor of the Proceedings of the VLDB Endowment (PVLDB). He received his Ph.D. from the University of Wisconsin, Madison, USA, and his Bachelor of Technology from the Indian Institute of Technology, Bombay, India. His research interests and publications span a variety of topics in data management.		* Bio: Divesh Srivastava is the head of Database Research at AT&T Labs-Research. He is a Fellow of the Association for Computing Machinery (ACM) and the managing editor of the Proceedings of the VLDB Endowment (PVLDB). He received his Ph.D. from the University of Wisconsin, Madison, USA, and his Bachelor of Technology from the Indian Institute of Technology, Bombay, India. His research interests and publications span a variety of topics in data management.

			* Lecture notes:

	== Week 14 - April 25th: Exploring Spatio-Temporal Data -- Invited lecture by Dr. Harish Doraiswamy (NYU CDS) ==		== Week 14 - April 25th: Exploring Spatio-Temporal Data -- Invited lecture by Dr. Harish Doraiswamy (NYU CDS) ==

Juliana: /* Week 12 - April 11th: Visualization: Using D3 -- Invited lecture by Bowen Yu */

2016-04-18T13:28:48Z

Week 12 - April 11th: Visualization: Using D3 -- Invited lecture by Bowen Yu

← Older revision		Revision as of 13:28, 18 April 2016
Line 132:		Line 132:

	* Lecture notes and lab:		* Lecture notes and lab:
	** http://vgc.poly.edu/~juliana/courses/BigData2016/Lectures/vis-d3.pdf		** UPDATED: http://vgc.poly.edu/~juliana/courses/BigData2016/Lectures/vis-d3_v2.1.pdf


	== Week 13 - April 18th: Data quality: the other face of big data - Invited lecture by Dr. Divesh Srivastava, AT&T Research ==		== Week 13 - April 18th: Data quality: the other face of big data - Invited lecture by Dr. Divesh Srivastava, AT&T Research ==

Juliana at 00:12, 18 April 2016

2016-04-18T00:12:52Z

← Older revision		Revision as of 00:12, 18 April 2016
Line 135:		Line 135:


	== Week 13 - April 18th: Data ~~Cleaning~~ - Invited lecture by Dr. Divesh Srivastava, AT&T Research ==		== Week 13 - April 18th: Data quality: the other face of big data - Invited lecture by Dr. Divesh Srivastava, AT&T Research ==

			* Abstract: In our Big Data era, data is being generated, collected and analyzed at an unprecedented scale, and data-driven decision making is sweeping through all aspects of society. Recent studies have shown that poor quality data is prevalent in large databases and on the Web. Since poor quality data can have serious consequences on the results of data analyses, the importance of veracity, the fourth “V” of big data is increasingly being recognized. In this talk, we highlight the substantial challenges that the first three “V”s, volume, velocity and variety, bring to dealing with veracity in big data. Due to the sheer volume and velocity of data, one needs to understand and (possibly) repair erroneous data in a scalable and timely manner. With the variety of data, often from a diversity of sources, data quality rules cannot be specified a priori; one needs to let the “data to speak for itself” in order to discover the semantics of the data. This talk presents recent results that are relevant to big data quality management, focusing on the two major dimensions of (i) discovering quality issues from the data itself, and (ii) trading-off accuracy vs efficiency.

			* Bio: Divesh Srivastava is the head of Database Research at AT&T Labs-Research. He is a Fellow of the Association for Computing Machinery (ACM) and the managing editor of the Proceedings of the VLDB Endowment (PVLDB). He received his Ph.D. from the University of Wisconsin, Madison, USA, and his Bachelor of Technology from the Indian Institute of Technology, Bombay, India. His research interests and publications span a variety of topics in data management.

Juliana at 20:36, 11 April 2016

2016-04-11T20:36:51Z

← Older revision		Revision as of 20:36, 11 April 2016
Line 113:		Line 113:
	** See quizzes on [http://www.newgradiance.com Gradiance] -- Distance measures and document similarity.		** See quizzes on [http://www.newgradiance.com Gradiance] -- Distance measures and document similarity.

	== Week 11 - April 4th: Large-Scale Visualization -- -- Invited lecture by Professor Claudio Silva ==		== Week 11 - April 4th: Large-Scale Visualization -- Invited lecture by Professor Claudio Silva ==

	* Lecture notes:		* Lecture notes:
Line 129:		Line 129:
	**http://vgc.poly.edu/~juliana/courses/BigData2016/Lectures/visualization/movies/SevereTstorm.mov		**http://vgc.poly.edu/~juliana/courses/BigData2016/Lectures/visualization/movies/SevereTstorm.mov

	== Week 12 - April 11th: Exploring Spatio-Temporal Data -- Invited lecture by Dr. Harish Doraiswamy (NYU CDS) ==		== Week 12 - April 11th: Visualization: Using D3 -- Invited lecture by Bowen Yu ==

			* Lecture notes and lab:
			** http://vgc.poly.edu/~juliana/courses/BigData2016/Lectures/vis-d3.pdf


			== Week 13 - April 18th: Data Cleaning - Invited lecture by Dr. Divesh Srivastava, AT&T Research ==


			== Week 14 - April 25th: Exploring Spatio-Temporal Data -- Invited lecture by Dr. Harish Doraiswamy (NYU CDS) ==

	* Lecture notes:		* Lecture notes:
Line 137:		Line 146:
	** https://github.com/ViDA-NYU/aws_taxi		** https://github.com/ViDA-NYU/aws_taxi

	~~== Week 13 - April 18th: Data Cleaning - Invited lecture by Dr. Divesh Srivastava, AT&T Research ==~~


	== Week 14 - ~~April 25th~~: Association Rules ==
			== Week 15 - May 2: Association Rules ==

	* Lecture notes:		* Lecture notes:
Line 155:		Line 164:
	* Homework Assignment		* Homework Assignment
	** See quiz on [http://www.newgradiance.com Gradiance] -- Association Rules.		** See quiz on [http://www.newgradiance.com Gradiance] -- Association Rules.

	~~== Week 15 - May 2: Graph Analysis ==~~

	* '''Lecture notes:''' http://vgc.poly.edu/~juliana/courses/BigData2015/Lectures/graph-algos.pdf

	* Required Reading: Data-Intensive Text Processing with MapReduce. Chapters 5 -- Graph Algorithms

Juliana: /* Week 11 - April 4th: Large-Scale Visualization -- -- Invited lecture by Professor Claudio Silva */

2016-04-11T20:32:08Z

Week 11 - April 4th: Large-Scale Visualization -- -- Invited lecture by Professor Claudio Silva

← Older revision		Revision as of 20:32, 11 April 2016
Line 116:		Line 116:

	* Lecture notes:		* Lecture notes:
			** http://vgc.poly.edu/~juliana/courses/BigData2016/Lectures/visualization/intro-to-visualization.pdf
			** http://vgc.poly.edu/~juliana/courses/BigData2016/Lectures/visualization/Plotting1.pdf
			** http://vgc.poly.edu/~juliana/courses/BigData2016/Lectures/visualization/Plotting2.pdf
			** http://vgc.poly.edu/~juliana/courses/BigData2016/Lectures/visualization/PlottingNotes.pdf
			** http://vgc.poly.edu/~juliana/courses/BigData2016/Lectures/visualization/Tufte.pdf


			* Videos:
			**http://vgc.poly.edu/~juliana/courses/BigData2016/Lectures/visualization/movies/biopathways.mov
			**http://vgc.poly.edu/~juliana/courses/BigData2016/Lectures/visualization/movies/VisTrailsForParaView_Small.mov
			**http://vgc.poly.edu/~juliana/courses/BigData2016/Lectures/visualization/movies/defog-1150.mov
			**http://vgc.poly.edu/~juliana/courses/BigData2016/Lectures/visualization/movies/SevereTstorm.mov

	== Week 12 - April 11th: Exploring Spatio-Temporal Data -- Invited lecture by Dr. Harish Doraiswamy (NYU CDS) ==		== Week 12 - April 11th: Exploring Spatio-Temporal Data -- Invited lecture by Dr. Harish Doraiswamy (NYU CDS) ==

Juliana: /* Week 11 - April 4th: Large-Scale Visualization -- -- Invited lecture by Professor Claudio Silva */

2016-04-11T20:27:27Z

Week 11 - April 4th: Large-Scale Visualization -- -- Invited lecture by Professor Claudio Silva

← Older revision		Revision as of 20:27, 11 April 2016
Line 115:		Line 115:
	== Week 11 - April 4th: Large-Scale Visualization -- -- Invited lecture by Professor Claudio Silva ==		== Week 11 - April 4th: Large-Scale Visualization -- -- Invited lecture by Professor Claudio Silva ==

			* Lecture notes:

	== Week 12 - April 11th: Exploring Spatio-Temporal Data -- Invited lecture by Dr. Harish Doraiswamy (NYU CDS) ==		== Week 12 - April 11th: Exploring Spatio-Temporal Data -- Invited lecture by Dr. Harish Doraiswamy (NYU CDS) ==

Juliana: /* Week 14 - April 25th: Association Rules */

2016-04-04T18:19:24Z

Week 14 - April 25th: Association Rules

← Older revision		Revision as of 18:19, 4 April 2016
Line 141:		Line 141:

	* Homework Assignment		* Homework Assignment
	** See ~~quizes~~ on [http://www.newgradiance.com Gradiance] -- ~~Distance measures and document similarity~~.		** See quiz on [http://www.newgradiance.com Gradiance] -- Association Rules.



	== Week 15 - May 2: Graph Analysis ==		== Week 15 - May 2: Graph Analysis ==