Course Project: Wikipedia Analysis - Revision history

Juliana: /* Submitting Code and Results */

2012-12-10T18:05:50Z

Submitting Code and Results

← Older revision		Revision as of 18:05, 10 December 2012
Line 184:		Line 184:
	** A discussion of the performance of your code, e.g., how long did it take to run the code? Which optimizations you used, if any?		** A discussion of the performance of your code, e.g., how long did it take to run the code? Which optimizations you used, if any?
	** Discuss any experiences you think will be useful to share with the class		** Discuss any experiences you think will be useful to share with the class

			* For Phase II, you should add the results to the Google Doc you submitted in Phase I. You should also add your code to GitHub -- you can use the same repository as in Phase I


	* You will present your findings during class on Nov 26th. You will have between 10-15min for your presentation.		* You will present your findings during class on Nov 26th. You will have between 10-15min for your presentation.

Juliana at 18:03, 10 December 2012

2012-12-10T18:03:53Z

← Older revision		Revision as of 18:03, 10 December 2012
Line 154:		Line 154:
	*** One of the groups had trouble with publicly available parsers. Here's the parser they are currently using and seems to work properly: Sweble -- http://sweble.org		*** One of the groups had trouble with publicly available parsers. Here's the parser they are currently using and seems to work properly: Sweble -- http://sweble.org

	== Phase 2: Data Analysis --- Due December ~~10th~~==		== Phase 2: Data Analysis --- Due December 17th before class!==

	Given the data products derived in phase 1, all groups will:		Given the data products derived in phase 1, all groups will:

Juliana: /* Phase 1: Data pre-processing --- Due November 22nd */

2012-11-30T21:38:26Z

Phase 1: Data pre-processing --- Due November 22nd

← Older revision		Revision as of 21:38, 30 November 2012
Line 127:		Line 127:
	* Viren Sawant (group without name): https://docs.google.com/document/d/1t3i5dCa37kuCrMXjI1-uSnqlYlKp-NmRqbLf--yHLmA/edit		* Viren Sawant (group without name): https://docs.google.com/document/d/1t3i5dCa37kuCrMXjI1-uSnqlYlKp-NmRqbLf--yHLmA/edit
	** Project members: Aman Luthra (aluthr01@students.poly.edu) Viren Sawant (vsawan01@students.poly.edu) Ankit Kapadia (akapad01@students.poly.edu)		** Project members: Aman Luthra (aluthr01@students.poly.edu) Viren Sawant (vsawan01@students.poly.edu) Ankit Kapadia (akapad01@students.poly.edu)
	** Output: ~~missing~~		** Output: S3n://task3group2output/output

	==== '''Task 4: ''' ====		==== '''Task 4: ''' ====

Juliana at 16:40, 30 November 2012

2012-11-30T16:40:18Z

← Older revision		Revision as of 16:40, 30 November 2012
Line 160:		Line 160:
	* Count number of pages that have an infobox, and the number of infoboxes that do not use a template		* Count number of pages that have an infobox, and the number of infoboxes that do not use a template
	* Group pages according to the kind of infobox they contain		* Group pages according to the kind of infobox they contain
	* Count the number of tables in all the Wikipedia pages
	* Build a histogram that shows the distribution of cross-language links across different languages, i.e., for each language show the total number of links for that language		* Build a histogram that shows the distribution of cross-language links across different languages, i.e., for each language show the total number of links for that language
	* Build a histogram that shows the recency of the pages, i.e., the histogram should how many pages have been modified on a particular date. You can use the last date of change for the pages.		* Build a histogram that shows the recency of the pages, i.e., the histogram should how many pages have been modified on a particular date. You can use the last date of change for the pages.
Line 168:		Line 167:
	* Some pages are assigned to multiple categories. Build a histogram that shows the number of pages that have 1, 2, 3, ..., n categories.		* Some pages are assigned to multiple categories. Build a histogram that shows the number of pages that have 1, 2, 3, ..., n categories.
	* Compute the word co-occurrence matrix (i.e., number of times word w_i occurs with word w_j within an article)		* Compute the word co-occurrence matrix (i.e., number of times word w_i occurs with word w_j within an article)

			You don't have to do:
			* Count the number of tables in all the Wikipedia pages

	Each group will hand in a report with the results of the analysis tasks. There are different ways to present these results. You are free to use your creativity to come up with visualizations that make it easier to convey the results.		Each group will hand in a report with the results of the analysis tasks. There are different ways to present these results. You are free to use your creativity to come up with visualizations that make it easier to convey the results.

Juliana: /* Task/Group Assignment */

2012-11-30T15:23:23Z

Task/Group Assignment

← Older revision		Revision as of 15:23, 30 November 2012
Line 114:		Line 114:
	==== '''Task 2: ''' ====		==== '''Task 2: ''' ====
	* Titans: https://docs.google.com/open?id=0B3e0J2n8r2MZaERFQjRtMU5FQUE		* Titans: https://docs.google.com/open?id=0B3e0J2n8r2MZaERFQjRtMU5FQUE
	** Project members: (tprive02@students.poly.edu) Mihir Patel (mpatel21@students.poly.edu) Sunil Gunisetty (sgunis01@students.poly.edu)		** Project members: Parth Trivedi (tprive02@students.poly.edu) Mihir Patel (mpatel21@students.poly.edu) Sunil Gunisetty (sgunis01@students.poly.edu)
			** Output: s3n://mpatel21/wikisearch/result/task2
	* Avishkar Nikum: https://docs.google.com/document/d/1h9_Giapd0gsZ2iTZ7W0ZJK1WBI9oBB6PJ2C1xHvlylQ/edit		* Avishkar Nikum: https://docs.google.com/document/d/1h9_Giapd0gsZ2iTZ7W0ZJK1WBI9oBB6PJ2C1xHvlylQ/edit
	** Project members: Avishkar Nikum (anikum01@students.poly.edu) Akshat Amritkar (a_akshat@yahoo.com) Zijian Wang (zwang17@students.poly.edu)		** Project members: Avishkar Nikum (anikum01@students.poly.edu) Akshat Amritkar (a_akshat@yahoo.com) Zijian Wang (zwang17@students.poly.edu)
	** Output: s3n://big.data.out/outs/		** Output: s3n://big.data.out/outs/

	==== '''Task 3:''' ====		==== '''Task 3:''' ====
Line 126:		Line 127:
	* Viren Sawant (group without name): https://docs.google.com/document/d/1t3i5dCa37kuCrMXjI1-uSnqlYlKp-NmRqbLf--yHLmA/edit		* Viren Sawant (group without name): https://docs.google.com/document/d/1t3i5dCa37kuCrMXjI1-uSnqlYlKp-NmRqbLf--yHLmA/edit
	** Project members: Aman Luthra (aluthr01@students.poly.edu) Viren Sawant (vsawan01@students.poly.edu) Ankit Kapadia (akapad01@students.poly.edu)		** Project members: Aman Luthra (aluthr01@students.poly.edu) Viren Sawant (vsawan01@students.poly.edu) Ankit Kapadia (akapad01@students.poly.edu)
			** Output: missing

	==== '''Task 4: ''' ====		==== '''Task 4: ''' ====

Juliana: /* Task 1: */

2012-11-29T04:19:16Z

Task 1:

← Older revision		Revision as of 04:19, 29 November 2012
Line 110:		Line 110:
	*Diǎosī: https://docs.google.com/document/d/1U2SgYDVU4XRBmZrGijdF_AaJ7C7yUr5Gu3H8HOAq9UY/edit (Zhe Zhou conanchou1412@gmail.com)		*Diǎosī: https://docs.google.com/document/d/1U2SgYDVU4XRBmZrGijdF_AaJ7C7yUr5Gu3H8HOAq9UY/edit (Zhe Zhou conanchou1412@gmail.com)
	** Project members: Zhe Zhou (conanchou1412@gmail.com), Kan Xiao (canly.xiao@gmail.com, 0464716)		** Project members: Zhe Zhou (conanchou1412@gmail.com), Kan Xiao (canly.xiao@gmail.com, 0464716)
			** Output: https://s3.amazonaws.com/diaosi-mapreduce/hadoop_data_final/part-00000

	==== '''Task 2: ''' ====		==== '''Task 2: ''' ====

Juliana: /* Task 3: */

2012-11-29T03:35:51Z

Task 3:

← Older revision		Revision as of 03:35, 29 November 2012
Line 121:		Line 121:
	* Team Indexers: https://docs.google.com/folder/d/0B5t0jWjSi7neSjVsOENiTW5Oa3M/edit		* Team Indexers: https://docs.google.com/folder/d/0B5t0jWjSi7neSjVsOENiTW5Oa3M/edit
	** Project members: Juan Rodriguez (jrodri04@students.poly.edu) Qi Wang (edwinj619@gmail.com) Ching-Che Chang (cchang08@students.poly.edu)		** Project members: Juan Rodriguez (jrodri04@students.poly.edu) Qi Wang (edwinj619@gmail.com) Ching-Che Chang (cchang08@students.poly.edu)
	** Output:		** Output: '''single file:''' s3n://indexers/output/All/part-00000.bz2 '''multiple files:''' s3n://indexers/output/All/task3-*.bz2
	single file: s3n://indexers/output/All/part-00000.bz2
	multiple files: s3n://indexers/output/All/task3-*.bz2

	* Viren Sawant (group without name): https://docs.google.com/document/d/1t3i5dCa37kuCrMXjI1-uSnqlYlKp-NmRqbLf--yHLmA/edit		* Viren Sawant (group without name): https://docs.google.com/document/d/1t3i5dCa37kuCrMXjI1-uSnqlYlKp-NmRqbLf--yHLmA/edit

Juliana: /* Task 3: */

2012-11-29T03:35:02Z

Task 3:

← Older revision		Revision as of 03:35, 29 November 2012
Line 121:		Line 121:
	* Team Indexers: https://docs.google.com/folder/d/0B5t0jWjSi7neSjVsOENiTW5Oa3M/edit		* Team Indexers: https://docs.google.com/folder/d/0B5t0jWjSi7neSjVsOENiTW5Oa3M/edit
	** Project members: Juan Rodriguez (jrodri04@students.poly.edu) Qi Wang (edwinj619@gmail.com) Ching-Che Chang (cchang08@students.poly.edu)		** Project members: Juan Rodriguez (jrodri04@students.poly.edu) Qi Wang (edwinj619@gmail.com) Ching-Che Chang (cchang08@students.poly.edu)
	** Output: ~~S3N~~://~~jcr~~.~~personal~~/~~data~~/*		** Output:
			single file: s3n://indexers/output/All/part-00000.bz2
			multiple files: s3n://indexers/output/All/task3-*.bz2

	* Viren Sawant (group without name): https://docs.google.com/document/d/1t3i5dCa37kuCrMXjI1-uSnqlYlKp-NmRqbLf--yHLmA/edit		* Viren Sawant (group without name): https://docs.google.com/document/d/1t3i5dCa37kuCrMXjI1-uSnqlYlKp-NmRqbLf--yHLmA/edit

Juliana: /* Task 4: */

2012-11-29T03:16:34Z

Task 4:

← Older revision		Revision as of 03:16, 29 November 2012
Line 129:		Line 129:
	*Team-kepaassafaidladkahai: did not complete		*Team-kepaassafaidladkahai: did not complete
	*A.F.K: https://docs.google.com/document/d/10Sx7VJF0rNveeD10aTOvJlLoE2gjfoIobVxGnmpM_3c/edit (Kien Pham kienpt.vie@gmail.com; Tuan-Anh Hoang-Vu hvtuananh@gmail.com; Fernando Seabra Chirigati fernando.chirigati@gmail.com)		*A.F.K: https://docs.google.com/document/d/10Sx7VJF0rNveeD10aTOvJlLoE2gjfoIobVxGnmpM_3c/edit (Kien Pham kienpt.vie@gmail.com; Tuan-Anh Hoang-Vu hvtuananh@gmail.com; Fernando Seabra Chirigati fernando.chirigati@gmail.com)
	* '''Output:'''		** '''Output:'''
	'''(1433 output files--with xxxx from 0000 to 1432)''' s3n://cs9223-results/2012-11-21-074821		'''(1433 output files--with xxxx from 0000 to 1432)''' s3n://cs9223-results/2012-11-21-074821
	https://s3.amazonaws.com/cs9223-results/2012-11-21-074821/part-0xxxx		https://s3.amazonaws.com/cs9223-results/2012-11-21-074821/part-0xxxx

Juliana: /* Task 3: */

2012-11-29T03:15:49Z

Task 3:

← Older revision		Revision as of 03:15, 29 November 2012
Line 123:		Line 123:
	** Output: S3N://jcr.personal/data/*		** Output: S3N://jcr.personal/data/*

	* Viren Sawant (group without name): https://docs.google.com/document/d/1t3i5dCa37kuCrMXjI1-uSnqlYlKp-NmRqbLf--yHLmA/edit (Aman Luthra; Viren Sawant; Ankit Kapadia)		* Viren Sawant (group without name): https://docs.google.com/document/d/1t3i5dCa37kuCrMXjI1-uSnqlYlKp-NmRqbLf--yHLmA/edit
			** Project members: Aman Luthra (aluthr01@students.poly.edu) Viren Sawant (vsawan01@students.poly.edu) Ankit Kapadia (akapad01@students.poly.edu)

	==== '''Task 4: ''' ====		==== '''Task 4: ''' ====