Big Data 2015: Final Project - Revision history

Juliana: /* Weather data */

2015-04-20T18:28:29Z

Weather data

← Older revision		Revision as of 18:28, 20 April 2015
Line 58:		Line 58:

	===Weather data===		===Weather data===
	* http://www.ncdc.noaa.gov/data-access/land-based-station-data/land-based-datasets ~~-- choose~~ either "Surface Data, Global Summary of the Day", or "Surface Data, Hourly Global" for a more detailed analysis. You can choose NY state, and ~~then~~ select the "John F Kennedy International Airport" station.		* http://www.ncdc.noaa.gov/data-access/land-based-station-data/land-based-datasets
	* http://www7.ncdc.noaa.gov/CDO/dataproduct -- select "Surface Data, Hourly Global", and then when it comes to select the ~~region~~, ~~choose NY and the three main stations (~~Central Park, JFK and LaGuardia).		* http://www7.ncdc.noaa.gov/CDO/dataproduct
			** Select either "Surface Data, Global Summary of the Day", or "Surface Data, Hourly Global" for a more detailed analysis.
			** You can then choose NY state, and select the "John F Kennedy International Airport" station (or all the stations, Central Park, JFK and LaGuardia).

	===Property and Construction data===		===Property and Construction data===

Juliana: /* Weather data */

2015-04-20T18:25:19Z

Weather data

← Older revision		Revision as of 18:25, 20 April 2015
Line 58:		Line 58:

	===Weather data===		===Weather data===
	* http://www.ncdc.noaa.gov/data-access/land-based-station-data/land-based-datasets		* http://www.ncdc.noaa.gov/data-access/land-based-station-data/land-based-datasets -- choose either "Surface Data, Global Summary of the Day", or "Surface Data, Hourly Global" for a more detailed analysis. You can choose NY state, and then select the "John F Kennedy International Airport" station.
	* http://www7.ncdc.noaa.gov/CDO/dataproduct -- select "Surface Data, Hourly Global", and then when it comes to select the region, choose NY and the three main stations (Central Park, JFK and LaGuardia).		* http://www7.ncdc.noaa.gov/CDO/dataproduct -- select "Surface Data, Hourly Global", and then when it comes to select the region, choose NY and the three main stations (Central Park, JFK and LaGuardia).

Juliana at 03:09, 6 April 2015

2015-04-06T03:09:51Z

← Older revision		Revision as of 03:09, 6 April 2015
Line 86:		Line 86:
	* You should describe the experimental setup (cluster configuration, number of mappers/reducers, tools you used) as well as report on the performance of your approach (e.g., report the running times of the scripts) and any optimizations you applied to speed up your code.		* You should describe the experimental setup (cluster configuration, number of mappers/reducers, tools you used) as well as report on the performance of your approach (e.g., report the running times of the scripts) and any optimizations you applied to speed up your code.
	* ''You should describe the individual contributions of each of the project's members.''		* ''You should describe the individual contributions of each of the project's members.''

			== Some Notes ==
			* For your analyses, it may be useful to have the fare and trip files merged. You already know how to do this ;-)
			* If you want to restrict your analyses to only consider trips that start or end in Manhattan, need to obtain the shape files for Manhattan and check whether the Lat/Long for the trip start (or end) are in the polygon defined by the shape files. Here are some links where you can find shape files:
			** NYC Neighborhoods shapefile in geojson: http://nycdata.pediacities.com/dataset?tags=neighborhoods
			** NYC ZIP: http://nycdata.pediacities.com/dataset/nyc-zip-code-tabulation-areas
			* To avoid float precision issues, convert money amounts to cents
			* Note that datetime is local datetime (with day-light saving). You can compute the dropoff_datetime by adding pickup_datetime to trip_time_in_secs -- this will help deal with changes in time.
			* Payments in cash often do not have a tip. Go figure...
			* Distances reported are in miles

Juliana: /* Project Mechanics */

2015-03-26T16:31:40Z

Project Mechanics

← Older revision		Revision as of 16:31, 26 March 2015
Line 75:		Line 75:
	* April 3: Submit the form with the information for your group. In the Google Doc, indicate your choice for the project, the data sets you will use, the tasks you will carry out, and a proposed timeline with weekly milestones.		* April 3: Submit the form with the information for your group. In the Google Doc, indicate your choice for the project, the data sets you will use, the tasks you will carry out, and a proposed timeline with weekly milestones.
	* April 10: Submit a status report describing any issues you encountered and updates to your initial plan.		* April 10: Submit a status report describing any issues you encountered and updates to your initial plan.
	* April 17: Submit a status report with preliminary results.		* April 20: Submit a status report with preliminary results.
	* May 11th: Final project report due.		* May 11th: Final project report due.
	* May 11th,18th: Project presentation: each group will present their results to the class. Each student will grade all the presentations (except, of course, their own ;-)		* May 11th,18th: Project presentation: each group will present their results to the class. Each student will grade all the presentations (except, of course, their own ;-)

Juliana at 16:29, 26 March 2015

2015-03-26T16:29:54Z

← Older revision		Revision as of 16:29, 26 March 2015
Line 49:		Line 49:
	*Fare data: http://chriswhong.com/wp-content/uploads/2014/06/nycTaxiFareData2013.torrent		*Fare data: http://chriswhong.com/wp-content/uploads/2014/06/nycTaxiFareData2013.torrent

	==Taxi data 2010-2013 ===		===Taxi data 2010-2013 ===
	*https://uofi.app.box.com/NYCtaxidata		*https://uofi.app.box.com/NYCtaxidata

Juliana at 16:29, 26 March 2015

2015-03-26T16:29:12Z

← Older revision		Revision as of 16:29, 26 March 2015
Line 9:		Line 9:
	In this project, you will create the infrastructure to automatically generate Fact Books for different years. The output could be a Web site where users can explore and compare the statistics for different years.		In this project, you will create the infrastructure to automatically generate Fact Books for different years. The output could be a Web site where users can explore and compare the statistics for different years.

	==Detecting Gentrification==		===Detecting Gentrification===
	Gentrification in NYC is a problem that has received substantial attention both in the media and in academia. See e.g.,		Gentrification in NYC is a problem that has received substantial attention both in the media and in academia. See e.g.,
	* http://www.nytimes.com/2015/02/25/opinion/the-gentrification-effect.html		* http://www.nytimes.com/2015/02/25/opinion/the-gentrification-effect.html

Juliana: /* Task */

2015-03-26T16:28:41Z

Task

← Older revision		Revision as of 16:28, 26 March 2015
Line 1:		Line 1:
	== Task ==		== Task ==
	~~You will analyze~~ NYC taxi data. ~~Different~~ groups will use the data to study ~~different~~ aspects urban life that can be detected from the taxi data and other related data sets..		By now, you are already an expert on the NYC taxi data. For the final project, different groups will use the data to study various aspects urban life that can be detected from the taxi data and other related data sets.

	You can select one or more of the areas below to focus your analysis on. While I provide suggestions for what you can look for, I expect you to use your creativity and go beyond those suggestions.		You can select one or more of the areas below to focus your analysis on. While I provide suggestions for what you can look for, I expect you to use your creativity and go beyond those suggestions.

Juliana: Created page with '== Task == You will analyze NYC taxi data. Different groups will use the data to study different aspects urban life that can be detected from the taxi data and other related dat…'

2015-03-26T15:53:41Z

Created page with '== Task == You will analyze NYC taxi data. Different groups will use the data to study different aspects urban life that can be detected from the taxi data and other related dat…'

New page

== Task ==
You will analyze NYC taxi data. Different groups will use the data to study different aspects urban life that can be detected from the taxi data and other related data sets..

You can select one or more of the areas below to focus your analysis on. While I provide suggestions for what you can look for, I expect you to use your creativity and go beyond those suggestions.
Besides exploring the taxi data, you must also use at least one additional data set in your analysis.

===Taxi Factbook===
Periodically, the Taxi & Limousine Commission releases a Fact Book (see http://www.nyc.gov/html/tlc/downloads/pdf/2014_taxicab_fact_book.pdf) where, for a given year, they list different statistics, e.g., the total number of medallions, the average number of miles traveled per year, number of passengers, etc.
In this project, you will create the infrastructure to automatically generate Fact Books for different years. The output could be a Web site where users can explore and compare the statistics for different years.

==Detecting Gentrification==
Gentrification in NYC is a problem that has received substantial attention both in the media and in academia. See e.g.,
* http://www.nytimes.com/2015/02/25/opinion/the-gentrification-effect.html
* http://opinionator.blogs.nytimes.com/2014/12/03/a-gentrification-story/?_r=0
* http://www.huffingtonpost.com/news/new-york-city-gentrification
* http://observer.com/2015/01/gentrification-may-be-complicated-but-its-not-a-myth-and-neither-is-displacement/
* http://socialwork.nyu.edu/news/2014/04/28/seminar-explores-complex-micro-and-macro-level-effects-of-gentrification-on-nyc-neighborhoods.html

Taxis can serve as sensors for economic activity in NYC, in that higher density of taxis in a region can serve as an indication of increased activity in that region.
Can we combine taxi data with other data sets better detect gentrification? For example:
* ACRIS (sales data): https://data.cityofnewyork.us/City-Government/ACRIS-Real-Property-Master/bnx9-e6tj
* Multi Agency Permits (including all applications for construction activity): https://data.cityofnewyork.us/City-Government/Multi-Agency-Permits/xfyi-uyt5

===Understanding taxi usage===
*Which neighborhoods are better served by taxis? How this correlates with the median household income per neighborhood ?
*How does taxi density vary over time? During the day? On weekdays vs. weekends and holidays?
*Are there regions where the taxi density is always high (or low)?
*What are the most popular destinations? Do these change over time? For example, summer vs the other seasons?
*What are the most popular trips (source, destination)?
*Is the number of trips and taxis affected by weather? E.g., are there fewer cabs when it is raining/snowing?

===Understanding taxi economics===
*How does revenue vary across neighborhoods and how does it correlate with the median household income in the neighborhood?
*How does revenue vary over time? Are the months or seasons when taxi companies make more (or less) money?
*How long do cab drives ride without passengers? How does this vary over time?
*Are revenues affected during major events? E.g., parades, presidential visits, storms

===Understanding driver behavior===
*How do different drivers work? Do drivers (or group of drivers) have a preferred neighborhood (or set of neighborhoods)? What does the pickup/dropoff distribution looks like? Does this preference change over time?
*Are there patterns shared among different drivers?
*Do some drivers get higher tips on average than others?
*Do some drivers take longer routes than others?
*Can you identify patterns for drivers that have higher income?
*How many hours do drivers often work each day/week? Are there outliers?

== Data sources ==
===Taxi data 2013===
*Trip data: http://chriswhong.com/wp-content/uploads/2014/06/nycTaxiTripData2013.torrent
*Fare data: http://chriswhong.com/wp-content/uploads/2014/06/nycTaxiFareData2013.torrent

==Taxi data 2010-2013 ===
*https://uofi.app.box.com/NYCtaxidata

===Census data===
*Demographics: http://www.nyc.gov/html/dcp/html/census/demo_tables_2010.shtml
*Income information: http://www.nyc.gov/html/dcp/html/census/socio_tables.shtml
* Shape files for census tracts: http://www.nyc.gov/html/dcp/html/bytes/districts_download_metadata.shtml (search for "tract")

===Weather data===
* http://www.ncdc.noaa.gov/data-access/land-based-station-data/land-based-datasets
* http://www7.ncdc.noaa.gov/CDO/dataproduct -- select "Surface Data, Hourly Global", and then when it comes to select the region, choose NY and the three main stations (Central Park, JFK and LaGuardia).

===Property and Construction data===
* ACRIS (sales data): https://data.cityofnewyork.us/City-Government/ACRIS-Real-Property-Master/bnx9-e6tj
* Multi Agency Permits (including all applications for construction activity): https://data.cityofnewyork.us/City-Government/Multi-Agency-Permits/xfyi-uyt5

== Project Mechanics ==
You should form a group with *at most 3 people*.
You have to use the Hadoop environment to carry out your analyses -- you can write mapreduce programs, use Pig, and any other tool that works on Hadoop.
Your code and scripts should be made available on GitHub and it should be '''reproducible''' -- you should include enough information so that others can reproduce what you did.
You will also maintain a GoogleDoc that you will share with me that describes your project, the questions you are investigating, and what you have done so far.
Please use this form to register your group and provide the information about your GitHub repo and Google Doc: https://docs.google.com/forms/d/1feAXUfUfgt2NgrHXf3xku3AdxPUWcfaxRv-h-cEfC1E/viewform?usp=send_form

Here are you milestones:
* April 3: Submit the form with the information for your group. In the Google Doc, indicate your choice for the project, the data sets you will use, the tasks you will carry out, and a proposed timeline with weekly milestones.
* April 10: Submit a status report describing any issues you encountered and updates to your initial plan.
* April 17: Submit a status report with preliminary results.
* May 11th: Final project report due.
* May 11th,18th: Project presentation: each group will present their results to the class. Each student will grade all the presentations (except, of course, their own ;-)

== Project Report ==
You can use your Google Doc as the starting point for your report.
* You should describe your experience, issues you encountered (e.g., dirty data) and how you dealt with them.
* You should report on and explain the findings of your analysis -- the use of insightful visualizations is highly encouraged.
* All results in your report should be reproducible -- the code/scripts you used should be made available together with instructions on how to run them to derive the results you obtained.
* You should describe the experimental setup (cluster configuration, number of mappers/reducers, tools you used) as well as report on the performance of your approach (e.g., report the running times of the scripts) and any optimizations you applied to speed up your code.
* ''You should describe the individual contributions of each of the project's members.''