Archive - Revision history

Remi at 19:44, 30 September 2013

2013-09-30T19:44:32Z

← Older revision		Revision as of 19:44, 30 September 2013
Line 27:		Line 27:
	The package would also have a command-line or graphical interface to add, remove or query files from the store.		The package would also have a command-line or graphical interface to add, remove or query files from the store.

	~~New~~ project is at [http://github.com/remram44/file_archive github.com/remram44/file_archive]		The new project is at [http://github.com/remram44/file_archive github.com/remram44/file_archive]

	== File store ==		== File store ==

	Just a plain filestore like Git's seems ~~file~~. There wouldn't be compression like Git's packs (although they are not really suitable for huge different files, but meant for similar small files). Sharing would still be possible with rsync (+ merging the databases) or anything else, as it's just about copying the files.		Just a plain filestore like Git's seems fine. There wouldn't be compression like Git's packs (although they are not really suitable for huge different files, but meant for similar small files). Sharing would still be possible with rsync (+ merging the databases) or anything else, as it's just about copying the files.

	~~Questions here:~~		It was decided to store directories as-is without deduplication (see [https://github.com/remram44/file_archive/issues/1 issue 1]).
	* Store a directory as a plain directory, with hash being some combination of the hashes of the contained files and filenames
	* Store a directory as ~~several objects, like Git does~~: ~~tree object referencing other trees and blobs~~

	== Which database? ==		== Which database? ==

			A database is used to store the metadata, and search for files matching a given condition on its metadata.

	=== MongoDB ===		=== MongoDB ===
	MongoDB was suggested by Fernando. Its role is precisely to store key-value pairs associated to an id.		MongoDB was suggested by Fernando during the meeting. Its role is precisely to store key-value pairs associated to an id.

	Pros:		Pros:
Line 59:		Line 59:
	* SQLite is bundled with Python		* SQLite is bundled with Python
	* SQL servers very common		* SQL servers very common
			* SQLite allows to save the database in a file, along with the store

	Cons:		Cons:
	* Join on key/value table		* Join on key/value table and general performance
	* Values with different types will need different tables? (e.g. for range queries to work)

	SELECT a.filehash, a.value AS value1, b.value AS value2		SELECT a.filehash, a.value AS value1, b.value AS value2
Line 94:		Line 94:
	* Moving the files is easy, it's just about adding the missing files to the destination. Tools like rsync could be used here (in a very dumb mode of operation).		* Moving the files is easy, it's just about adding the missing files to the destination. Tools like rsync could be used here (in a very dumb mode of operation).
	* Merging the databases is not complicated either, we just need to merge the databases, i.e. add the missing records to the other database (no update necessary).		* Merging the databases is not complicated either, we just need to merge the databases, i.e. add the missing records to the other database (no update necessary).

			== Open design issues ==

			See [https://github.com/remram44/file_archive/issues?labels=question&state=open the github issues marked 'question'].

Remi: /* Syncing repositories */

2013-09-26T16:43:59Z

Syncing repositories

← Older revision		Revision as of 16:43, 26 September 2013
Line 87:		Line 87:

	(see [http://sqlfiddle.com/#!1/15431/1/0 sqlfiddle])		(see [http://sqlfiddle.com/#!1/15431/1/0 sqlfiddle])

			== Syncing repositories ==

			Once files have been generated and stored, the user probably want to put them into another archive.

			* Moving the files is easy, it's just about adding the missing files to the destination. Tools like rsync could be used here (in a very dumb mode of operation).
			* Merging the databases is not complicated either, we just need to merge the databases, i.e. add the missing records to the other database (no update necessary).

Remi: /* SQL */

2013-09-26T15:59:30Z

SQL

← Older revision		Revision as of 15:59, 26 September 2013
Line 70:		Line 70:
	GROUP BY filehash		GROUP BY filehash

	(see [http://sqlfiddle.com/#!2/~~73024~~/1/0 sqlfiddle])		(see [http://sqlfiddle.com/#!2/1e627/5/0 sqlfiddle])

	=== PostgreSQL's hstore type ===		=== PostgreSQL's hstore type ===

Remi: Github link

2013-09-26T14:34:08Z

Github link

← Older revision		Revision as of 14:34, 26 September 2013
Line 27:		Line 27:
	The package would also have a command-line or graphical interface to add, remove or query files from the store.		The package would also have a command-line or graphical interface to add, remove or query files from the store.

	New project is at ~~<tt>vgc~~.~~poly~~.~~edu:src~~/file_archive~~</tt>~~		New project is at [http://github.com/remram44/file_archive github.com/remram44/file_archive]

	== File store ==		== File store ==

Remi at 21:47, 25 September 2013

2013-09-25T21:47:57Z

← Older revision		Revision as of 21:47, 25 September 2013
Line 70:		Line 70:
	GROUP BY filehash		GROUP BY filehash

	(see [http://sqlfiddle.com/#!2/73024/1/0 ~~jsfiddle~~])		(see [http://sqlfiddle.com/#!2/73024/1/0 sqlfiddle])

	=== PostgreSQL's hstore type ===		=== PostgreSQL's hstore type ===
Line 86:		Line 86:
	SELECT * FROM files WHERE metadata @> '"key1"=>"value1","key2"=>"value2"'::hstore;		SELECT * FROM files WHERE metadata @> '"key1"=>"value1","key2"=>"value2"'::hstore;

	(see [http://sqlfiddle.com/#!1/15431/1/0 ~~jsfiddle~~])		(see [http://sqlfiddle.com/#!1/15431/1/0 sqlfiddle])

Remi: Git link

2013-09-25T20:33:47Z

Git link

← Older revision		Revision as of 20:33, 25 September 2013
Line 26:		Line 26:

	The package would also have a command-line or graphical interface to add, remove or query files from the store.		The package would also have a command-line or graphical interface to add, remove or query files from the store.

			New project is at <tt>vgc.poly.edu:src/file_archive</tt>

	== File store ==		== File store ==

Remi: /* File store */ How to store directories

2013-09-25T18:37:38Z

File store: How to store directories

← Older revision		Revision as of 18:37, 25 September 2013
Line 30:		Line 30:

	Just a plain filestore like Git's seems file. There wouldn't be compression like Git's packs (although they are not really suitable for huge different files, but meant for similar small files). Sharing would still be possible with rsync (+ merging the databases) or anything else, as it's just about copying the files.		Just a plain filestore like Git's seems file. There wouldn't be compression like Git's packs (although they are not really suitable for huge different files, but meant for similar small files). Sharing would still be possible with rsync (+ merging the databases) or anything else, as it's just about copying the files.

			Questions here:
			* Store a directory as a plain directory, with hash being some combination of the hashes of the contained files and filenames
			* Store a directory as several objects, like Git does: tree object referencing other trees and blobs

	== Which database? ==		== Which database? ==

Remi: /* Interface */

2013-09-25T16:16:21Z

Interface

← Older revision		Revision as of 16:16, 25 September 2013
Line 16:		Line 16:
	* Use a database to associate hash filename (upstream modules hash?) to metadata: vistrails parameter, (execution info), custom metadata		* Use a database to associate hash filename (upstream modules hash?) to metadata: vistrails parameter, (execution info), custom metadata
	* Request: make this separate from VisTrails (and used by the new archive package) so that it can be used directly by other code, and used to find files in the store		* Request: make this separate from VisTrails (and used by the new archive package) so that it can be used directly by other code, and used to find files in the store

			== Interface ==

			* Store a file: gets a file and a dictionary of metadata, hashes it, puts it in the store with the hash name, and creates a database entry.
			* Remove a file: gets a hash, removes that file and its associated metadata
			* Query: searches for hashes matching conditions on metadata from the database

			On top of this file-storing package, the VisTrails package would be built. It would provide modules like the persistence package, to add a file or directory in storage or retrieve one, a graphical querying UI, and a way to get the file from a module in a specific workflow version.

			The package would also have a command-line or graphical interface to add, remove or query files from the store.

			== File store ==

			Just a plain filestore like Git's seems file. There wouldn't be compression like Git's packs (although they are not really suitable for huge different files, but meant for similar small files). Sharing would still be possible with rsync (+ merging the databases) or anything else, as it's just about copying the files.

	== Which database? ==		== Which database? ==

Remi at 16:08, 25 September 2013

2013-09-25T16:08:44Z

← Older revision		Revision as of 16:08, 25 September 2013
Line 65:		Line 65:

	SELECT * FROM files WHERE metadata @> '"key1"=>"value1","key2"=>"value2"'::hstore;		SELECT * FROM files WHERE metadata @> '"key1"=>"value1","key2"=>"value2"'::hstore;

			(see [http://sqlfiddle.com/#!1/15431/1/0 jsfiddle])

Remi: /* Which database? */ Pros and cons

2013-09-25T16:06:32Z

Which database?: Pros and cons

← Older revision		Revision as of 16:06, 25 September 2013
Line 21:		Line 21:
	=== MongoDB ===		=== MongoDB ===
	MongoDB was suggested by Fernando. Its role is precisely to store key-value pairs associated to an id.		MongoDB was suggested by Fernando. Its role is precisely to store key-value pairs associated to an id.

			Pros:
			* Built for schema-less storage
			* Supports range queries and such

	Cons:		Cons:
	* MongoDB required		* MongoDB required: big installation, needs server to be running
	* Reliability?		* Reliability?
	* An index would need to be created for each attribute		* An index would need to be created for each attribute (performance unknown)

	find({'key1': 'value1', 'key2': 'value2'})		find({'key1': 'value1', 'key2': 'value2'})
Line 31:		Line 35:
	=== SQL ===		=== SQL ===
	This would required a filename/key/value table (or even, one per value type). JOINing could be painful.		This would required a filename/key/value table (or even, one per value type). JOINing could be painful.

			Pros:
			* SQLite is bundled with Python
			* SQL servers very common

	Cons:		Cons:
	* Join on key/value table		* Join on key/value table
	* Values with different types will need different tables?		* Values with different types will need different tables? (e.g. for range queries to work)

	SELECT a.filehash, a.value AS value1, b.value AS value2		SELECT a.filehash, a.value AS value1, b.value AS value2
Line 48:		Line 56:

	hstore is a key-value store as a single value (i.e. in a column).		hstore is a key-value store as a single value (i.e. in a column).

			Pros:
			* Common SQL server

			Cons:
			* PostgreSQL requried: big installation, needs server to be running
			* Non-equality queries limited?

	SELECT * FROM files WHERE metadata @> '"key1"=>"value1","key2"=>"value2"'::hstore;		SELECT * FROM files WHERE metadata @> '"key1"=>"value1","key2"=>"value2"'::hstore;