Recently I've put a lot of time into the Overgrowth file system, so I would like to share the best practices I've found for efficient file management on Mac, Windows and Linux. This is a pretty broad topic, so I'll break it into parts. In this first part, we can look at file paths. That might sound like a trivial topic, but there are actually a surprising number of edge cases to worry about! Here are all of the file path concerns I've run into, from simplest to most complicated.

Slash Direction

On Unix-based systems like Linux or Mac OS X, file path directories are separated by the forward slash character '/', like this:
"/Users/David/Wolfire/Project/Overgrowth.xcodeproj". Conversely, Windows file path directories are separated with the backslash character '', like this: "C:\Users\David\Wolfire\Project\Overgrowth.vcproj". If we try to open a file on a Unix-based system using the Windows backslash, it will report that the file can't be found!

However, if we open a file on a Windows system using the Unix forward slash, it will work just fine. Since forward slashes work on Windows, Mac OS X, and Linux, it's safer to always use them instead of backslashes, and convert the backslashes to forward slashes whenever we request a file path from Windows.

Case Sensitivity

Most Mac and Windows filesystems are not case-sensitive, so if we have a file at "Data/flame.tga", then we can open it with a request for "Data/Flame.tga" or even "DaTa/FlAmE.TGa". If our development machine is not case-sensitive, it's easy for case-incorrect file paths to slip in unnoticed. Then when someone tries to play our game on a case-sensitive file system, it will fail!

There are several solutions to this problem. The simplest solution is to just make all of our file paths are lower-case, and enforce that in code (e.g. by displaying an error if our file-loading code sees an upper-case character). Another solution is to completely bypass the native filesystem, so it doesn't matter if it's case-sensitive or not. For example, we can store all of our files in a custom archive like a .zip file, and load it using something like the PhysicsFS library.

But what if we want to open loose files, and don't want to enforce lower-case? Then we can work around the case-sensitivity of the native filesystem. We can simulate case-insensitivity by walking through directory structures and doing case-insensitive comparisons (example code), and then it doesn't matter if our paths are case-correct.

Conversely, we can simulate case-sensitivity by double-checking every path with the native filesystem path, making sure that they match (example code). Overgrowth currently uses both methods: it will display an error message if the case is incorrect, but it will still work if we click through it.

Working Directories

Users will usually open our game using a shortcut: from the desktop, dock, start menu, or other launcher. If the user opens the desktop shortcut, then the game might try to load a file like "Data/Flame.tga", and report that "Users/USERNAME/Desktop/Data/Flame.tga" does not exist. Of course it doesn't, that's the wrong path!

This happens because shortcuts can change our working directory (the directory that is implicitly prepended to relative paths). Typically we want it to always be something like "Programs/APPNAME/", so that it looks for "Data/Flame.tga" in "Programs/APPNAME/Data/Flame.tga".

To make sure we always have the right working directory, we can manually find the path to the executable, and set the working directory using "chdir". On Windows we can find the executable path using GetModuleFileName(), which will give us something like: "C:\Program Files (x86)\Wolfire\Overgrowth\Overgrowth.exe", and then we can just cut off the end to get: "C:\Program Files (x86)\Wolfire\Overgrowth\", and set that to be our working directory.

On Mac or Linux we search the command-line arguments and $PATH to find the working directory, but it looks like readlink() might be a simpler way to do it (see this Stack Overflow thread).

Write Permissions

Once upon a time, we could just create saved-game files and config files within our game directory, and that was that. On modern operating systems, we can't do that anymore, because the OS does not typically give us permission to write to the program directories. Now each OS has a different location that it expects programs to write documents to. For example, on Mac OS X, these files are usually written to "Users/USERNAME/Library/Application Support/APPNAME/". On Windows, they go to "Users/USERNAME/Documents/APPNAME/". On Linux, they just go to "Users/.APPNAME/".

We can handle this by creating a string to store the write directory for the current OS, and prepending this string to the beginning of every file write that we perform. On Windows we can use SHGetFolderPath(... CSIDL_PERSONAL ...) to get "Users/USERNAME/Documents/", and add "Wolfire/Overgrowth/" to get the complete write string. Then if we want to write to "Data/config.txt", we just add "Users/USERNAME/Documents/Wolfire/Overgrowth/" to the front, to get the complete write path: "Users/USERNAME/Documents/Wolfire/Overgrowth/Data/config.txt"

On Mac we can get the "Users/USERNAME/" from getenv("HOME"), and then just add "Library/Application Support/Overgrowth/". On Linux we can also use getenv("HOME"), and just add ".overgrowth/".

Unicode Paths

It's easy to just ignore non-ASCII characters. If all our data files and directories just use simple English text then we should be safe, right? Wrong! Remember all those USERNAME macros we have to deal with to get write directories? Those could easily be something like 成龍. If we try to write to "Users/成龍/Documents/Wolfire/Overgrowth/config.txt" on Windows without dealing with Unicode, then we'll end up creating a whole new User directory called "??" as we write to "Users/??/Documents/Wolfire/Overgrowth/config.txt".

For those not familiar with Unicode, here's a very brief overview. Unicode is a text encoding that provides a unique index for every character in every language. For example, 成龍 is character #25104 followed by character #40845. Because there are so many unique characters, we need to have an array of 32-bit ints to store them, if we want exactly one character in each array slot. This encoding is called UTF-32 (short for "Unicode Transformation Format - 32-bit").

Conveniently, the ASCII characters are unchanged in Unicode, so strings like "Data/Flame.tga" or "Hello World!\n" are the same in ASCII and in Unicode. This means that if we have existing functions that convert backslashes to forward slashes or capitalize ASCII characters, they should still work pretty well with UTF-32.

So far this is pretty simple -- we just have to use int strings instead of char strings, or std::basic_strings instead of std::strings. No big deal! However, things get more complicated when passing these strings to and from the operating system calls. This is because none of the major operating systems actually accept or provide UTF-32 strings for file paths: Mac and Linux use UTF-8, and Windows uses UTF-16.

As you might have guessed, UTF-8 uses 8-bit values, and UTF-16 uses 16-bit values. But how can we fit a character like 成 (#25104) into an 8-bit value? We can't, it actually converts to a sequence of three UTF-8 values: (#230 #130 #132). Some characters use four UTF-8 values, and some only use one (namely, the 7-bit ASCII characters). In short, you can think of UTF-8 and UTF-16 as variable-length lossless compression formats for UTF-32. If you want to convert from one format to another in C++, UTF-8 CPP is a decent light-weight library.

In Overgrowth, we use UTF-32 strings to manipulate file path strings, and convert them to and from UTF-8 for Mac and Linux API calls, and to and from UTF-16 for Windows API calls. The Mac and Linux calls like fopen(), readpath(), and so on, are unchanged for UTF-8. In fact, they always use UTF-8, and ASCII users don't notice because the common characters are unchanged. The Windows calls, on the other hand, use variations like _wfopen() or GetModuleFileNameW() to specify that they use "wide chars", which are UTF-16 on Windows.

Conclusion

These are all of the file path issues that I've run into while working on Overgrowth. Planned topics for future posts include caching processed files, improving load speed, and choosing when to load files. Please let me know if there are any other topics you'd like to see, or if you've found better solutions to these file path issues!

This week I focused on internal engine technology again, and was planning to make a video anyway, but ended up programming all day instead. There will definitely be a video next week though!

Here are the changes for Alpha 179:

• Level pre-loading
  
• Background texture conversion
  
• Image sampler and heightmap caches
  
• New animation cache system
  
• Fix for strange particle colors on Red Shards
  

Don't forget that you can help support us, try out our weekly alphas, and chat with other preorderers in the Secret Preorder Forum by preordering Overgrowth. If you'd like to see real-time news about Overgrowth, you can follow me on Twitter at @wolfire.

I'm thinking that blog posts might be more appropriate than videos when I'm doing tech work -- would you be interested in blog posts about game file systems? It's not as flashy as graphics or physics, but arguably more important.

This week I focused on internal features, such as .zip file archive support, texture compression on background threads, and preparations for the upcoming Linux build! These features are all still in progress, though, and difficult to show in their current form, so I thought I should save them for an upcoming video.

This video shows off a new level I'm working on while attempting to refine the visuals. I would like to achieve finished-looking levels in the next few weeks so that I can focus more on level design and less on art assets.

I spent some time on new terrain textures and more experimentation with the detail objects system. I was trying to make the wooded area on this new map look more like a real forest floor with various debris and different plants. I also tried grouping a lot of smaller objects to a rock, and then copying out the group rather than spending a lot of time adding tiny unique details to each rock I place.

Getting the details right for each environment takes a lot of work, and each area seems a lot bigger once it is full of stuff, so I have been contemplating the idea of doing a few different missions on each environment for the final story mode. One way to keep that from becoming stale would be to change it a bit every time you visit, like having the mission take place at a different time of day or under different weather conditions.

If you liked this video, be sure to subscribe below.

Here is the new weekly Overgrowth alpha video!

Don't forget that you can help support us, try out our weekly alphas (such as the one in the video), and chat with other preorderers in the Secret Preorder Forum by preordering Overgrowth. If you'd like to see real-time news about Overgrowth, you can follow me on Twitter at @wolfire.

The features highlighted in the above video are as follows (as well as some that didn't make it into the video):

• Animated debug text using webkit
• Set up proper animatable wolf rig
• Started to overhaul level load system
• Updated angelscript to 2.22.1
• Fixed memory leak when saving shadow textures
• Fixed occasional crash on quit in angelscript destructor

Thanks as always for all the support! See you guys in IRC and the forums.

If you liked this video, be sure to subscribe to our YouTube channel.

Also, feel free to support us by preordering Overgrowth!