Accessing Wikipedia data on S3

From VistrailsWiki
Jump to navigation Jump to search

There are 2 ways to access the Wikipedia segments:

  • By HTTP. Here are the link to 27 files:

  • Through Hadoop on EC2. Hadoop supports access to S3 directly, so anyone with an access key and secret key configured in core-sites.xml will be able to access it. For example

bin/hadoop fs -ls s3n://cs9223/enwiki-20121001/

(s3n is the s3 native filesystem)

To access these files from any machine with hadoop installed, open core-site.xml and add the following:


Note: The property name should be fs.s3n for S3 native file system.

See for more information