Hello everyone, I’m in the process of setting up long term storage of a large amount of media files for my family. Since this is stuff that’s rarely accessed, I don’t need it to be always fully available, and I want to limit the actual transfer of files to a specific schedule.
Since knowing what’s already in storage is kinda non-trivial, are there any tools out there that allow me to browse which files are in cold storage, without the actual drives being attached / mounted? So only info like file location, name, metadata, perhaps icon, etc.
The transfer part I can manage to automate, as I can think of a few ways of allowing my users to “stage” pulling and pushing info from and to cold storage, and having the transfers be done on a fixed schedule like every two weeks or something.
Obviously self-hosted, open source stuff that’s not vibecoded is preferred.
Thanks in advance!
This is exactly what git-annex was built for:
https://git-annex.branchable.com/
use case: The Archivist
Bob has many drives to archive his data, most of them kept offline, in a safe place.
With git-annex, Bob has a single directory tree that includes all his files, even if their content is being stored offline. He can reorganize his files using that tree, committing new versions to git, without worry about accidentally deleting anything.
When Bob needs access to some files, git-annex can tell him which drive(s) they’re on, and easily make them available. Indeed, every drive knows what is on every other drive.
I have no need for it, but what a neat piece oft Software.
Just out of curiosit, what do you gain from unmounting the fs? Depending on your reasoning, it might he an option to use a zfs filesystem, set it up so automatic jobs rarely run and then set a somewhat agressive disk spin down policy. That way fs info would be handled by the arc, not the disks and they would only spin up when an actual file is accessed.
At least I assume it would work this way. Look-Ahead reads or your explorer generating previews, etc. might well make thta option fail.
Good question, as it’s not really obvious from the post. There’s simple stuff such as energy savings, preventing unneeded file indexes, preventing corruption, etc. The biggest reason is mostly situational to me, as I’ve taken over a bunch of drives from friends who renewed their own, which have been running in their respective NAS setups. The drives are definitely not new but have a reasonable remaining life span. I kinda realized that I don’t need the media files to be available at all times, so to squeeze the most out of the drives, I wanted to try this strategy.
I haven’t yet tried adjusting spin-down options as it’s more predictable when I “manually” spin the drive up and down. I’m afraid the random seeks will still spin up the drives several times a day and hurt them over time.
Not a GUI/webapp, but I like to use ncdu snapshots (“exports”). ncdu is an interactive file browser / disk usage analyzer with a terminal UI, which allow for convenient browsing but also, y’know, figuring out what to delete in case of storage cleanups.
creating a snapshot (“exporting”):
ncdu /pathyouwanttoscan -2 --one-file-system --compress -o snapshot_ncdu.json.zstd(
-2enables progress bar and error messages during the scan)reading (“importing”):
ncdu -f snapshot_ncdu.json.zstdLove ncdu, didn’t know it had this capability. Cool!
Before unmounting:
$ tree -afx /PATH/TO/VOLUME > file_listing.txtLater, when you want to search:
$ grep PATTERN file_listing.txtHaven’t used it yet, but I think git annex is built for this. Plan to do the same eventually. It lets you keep the index on hot storage.
