Archive

Matthias Troyer came to Poly to discuss his use of VisTrails and the problems he was facing with the persistence package.

Fernando, Juliana, Matthias and Remi met on 2013-09-24.

Summary

Persistence only used as a cache

• Can’t delete stuff; he deletes and recreates the whole store
• He wants to use it to archive correct result, without the other intermediate files that resulted from bogus workflow OR module code
• He wants to be able to find his files afterwards. Git revision hash + file reference = impracticable
• He doesn’t mind filenames being unreadable if he has some way of finding these from metadata (workflow name, vistrail query, or custom metadata from module code)

Conclusions:

• Drop git. If we are only going to use it for storage, and keep a separate database to map from ref uuid/upstream hash to object hash, commits and branches are useless (and are a nuisance because we can’t rewrite history)
• Use a flat object store with hashes (like git’s)
• Use a database to associate hash filename (upstream modules hash?) to metadata: vistrails parameter, (execution info), custom metadata
• Request: make this separate from VisTrails (and used by the new archive package) so that it can be used directly by other code, and used to find files in the store