Organizing a Game Development and Music Asset Library
Another thing I’ve been meaning to do for some time is organize all of my game development and music production assets. I’ve been looking for a good web-based file browser that has advanced search features (potentially with AI) that can be hosted in a docker. This would make life easier when searching for things. Unfortunately, there aren’t many choices, so maybe I will get around to programming one myself; but in the mean time, I need to organize the files.
First, I downloaded all of my assets from the different store front websites. (Itch.io, GameDevMarket, Fanatical, Humble Bundle, AAS, what I could extract from Unity, and Unreal, and Patreon).
Next, on the server, I created my assets folder, and created the root structure for all of the assets. I haven’t really planned everything out yet, but I figured that if I started at the top, I could slowly start nesting the proper way. (The /STORES/ directory is just for holding the zip files until I get them sorted).
./assets ├── 3d-printer ├── STORES ├── artwork ├── audio ├── brushes ├── fonts ├── software └── vst
Because I have a lot of assets, and I tend to “test” things out with bash during my process, I have a safe location for the original zip files so that I can recover if needed without having to download everything again.
According to ‘du’, the total size of my compressed asset collection is approximately 185GB.
du -hLx /mnt/user/storage/Original\ Downloaded\ Assets/ | awk ‘END{ print $1 }’
185G
Now, if you are like me, and you are also trying to organize your assets, I highly suggest you set up your collection storage like I did (with all of your original asset downloaded files in a separate location). It’s easy to identify missing files and manage what you’ve downloaded. For instance, I have about 130 purchases on Fanatical, and I have 130 downloads in my Fanatical folder. For someone like me, it just makes sense.
When ready, copy your assets into a temp folder where you can parse and organize your collection without modifying the originals.
rsync -arp /mnt/user/storage/Original\ Downloaded\ Assets/ /mnt/user/assets/STORES/
(Remember how linux works, commands almost always following the [command] [arguments] [source] [destination] methodology.). And, if you need to look up what arguments I’ve used, you can ‘man’ the command or use ‘–help’.
‘-a’ is archive mode; which preserves as many of the attributes possible
‘-r’ is recursive mode; it will go into sub-directories and copy files and directories as well
‘p’ is preserve permissions mode; this just keeps the permissions the same when copied. (Probably redundant with -a, but it’s a force of habit for me).
You can also use ‘–info=progress2’ if you want to see the approximate time remaining.
Now that there is a good copy of the library, the next step is to separate the most easily identifiable group first. I chose audio because they are simple to identify and most art-based assets don’t have audio files. I think the best starting group of file extensions is: wav, mid, midi, mp3, flac, acc, ogg, wma, and voc. Of course, if you need more, just add them in. This is just my list.
** because I know there are duplicates in my library, I do a simple find / delete to remove the files. It just so happens that Firefox appends a ‘(1)’ to the end of files that are downloaded twice (or that have the same name).
find ./STORES/ -type f -iname “(1)”
After inspecting the list to make sure that nothing is wrong, then I just add remove at the end:
find ./STORES/ -type f -iname “(1)” -exec rm {} ;
Now, this is also a smart time to standardize the names. A lot of times, these asset files don’t have a naming standard, so some things are all caps, some are camel case, some have spaces, while others use underscores or dashes. I like to remove all special characters with the exception of dash and dot. I also like to force everything to lowercase. It’s the easiest to work with at this moment. This bash line just finds all files within a directory (recursively), and then renames the file with the previously mentioned convention. (All lowercase and replaces spaces and underscores with a dash).
find . -type f -exec sh -c ’new_name=$(echo “${1##*/}” | tr “[:upper:]” “[:lower:]” | sed “s/[ _]/-/g; s/[^a-zA-Z0-9.-]//g”); dir_name=$(dirname “$1”); mv “$1” “$dir_name/$new_name”’ _ {} ;
This command will search all the zip files for these file extensions, if the zip file has this file type, then it will move the zip file to the specified location. This is probably one of my -more- favorite bash one-liners as it is very useful for looking in a zip file but not extracting it, and it is fairly quick.
find ./STORES/ -type f -iname “*.zip” -exec sh -c ‘unzip -l “{}” | grep -Ei “.(wav|mid|midi|mp3|flac|acc|ogg|wma|voc)$” && mv “{}” ./audio/’ ;
Next, I moved all of the stl files. (Same command, but this time I searched for ‘stl’ and ‘gcode’).
You can run this as many times as you’d like to get all of the desired zip files where they need to go. My goal is to just narrow everything down so that it’s easier to identify everything.
