Logo

Ljung

.dev
Tutorials

Hej!

Git is a great version control system, but can struggle with large project, notably for lack of handling/scaling with large binary files. Those projects often opt for a different solution, like PlasticSCM or Perforce, however those versioning system work in a vastly different way, with their own advantages and drawbacks.

Git however nowadays support blobs through Git LFS, which essentially store user-specified files outside the repo itself. Supposedly it's not a 100% solution to the problem, and GitHub has somewhat narrow limits for an Unreal project. This however led me to an idea for my specific use case. In essence, I:

  • Work on solo projects

  • Want to track revisions

  • Want to access my code on different machines

  • Open-source the codebase of my projects

  • Don't want to expose my content, which would make commercializing my open-source projects difficult

With this in mind I devised a structure and workflow to split content (essentially the Content folder but also some internal source material) and code/config (everything else).

Note: this workflow is still experimental and I'm testing it as I go through my solo projects.

Architecture

An entire project is composed of two separate Git repositories: the "Assets" and the "Main" repository. The "Assets" repo is in turn added as a Git submodule of the "Main" repo. This simply means that folder inside "Main" is considered an entirely separate repo and is not included in the changes of the "Main" repo; the "Main" repo simply tracks which revision of the "Assets" repo is in use (and commits that tracking).

Setup

Assets repo

The assets repo may live anywhere you like. For my purposes I put it on my NAS. My Unreal project is named "Cactus" and so my assets repo will be called "Cactus_Assets". From the directory where this repo will live run:

bash
1git init
2git branch -m master main
3git config --local receive.denyCurrentBranch updateInstead
4git lfs install
5git lfs track "*.uasset"
6git lfs track "*.umap"
7git add .gitattributes
8git commit -m "🎉 Init: setup"

This repo has to be non-bare to be used as a submodule. Because of this we also need to set the denyCurrentBranch policy in order to push from our Main repo later on. Since the repo is non-bare this setting will force the working tree to be clean. This shouldn't be a problem as long as you treat it as a bare repo and don't work directly from the assets repo.

You may also want to track additional file-types for LFS. For my scenario I store source files outside the Content folder, but if you store them together you might want to add ".fbx", ".png" etc.

Renaming branch is optional but I like to match it with how GitHub names it.

Main repo

Remove the Content folder if not already removed. My preference is also to keep the Unreal project in a separate folder inside the Main repo, which will have implications when we add the Assets repo submodule. My structure looks like this:

  • Cactus_Root

    • Cactus

      • Source

      • Cactus.sln

      • Cactus.uproject

      • etc.

    • SourceArt

    • etc.

From the root directory run:

bash
1git init
2git branch -m master main
3git submodule add -b main Y:\Archive\GameDev\Cactus_Assets .\Cactus\Content
4git config push.recurseSubmodules on-demand

Note that Y:\Archive\GameDev\Cactus_Assets is the full path to my Assets repo, and you have to rename "Cactus" in .\Cactus\Content to what your project is named.

Settings push.recurseSubmodules means that any commits not pushed on the Assets repo will be pushed before pushing Main repo commits. This is a best practice since the Main repo tracks and commits which revision of the Assets repo it is using, and thus the Assets repo commit has to be available before updating any remotes.

Workflow

Working with this setup is largely the same as a regular single repo. The difference is if you use Git commands from within the Content folder it will act upon the Assets repo, while anywhere else will act upon the Main repo.

Any changes to the Content folder should be committed before related changes in the Main repo are committed, to ensure interoperability stays consistent. As you make commits and update the Assets repo the Main repo will see changes to a file pointing to the Content directory. This file should be committed, and simply tracks the Assets repo commit being used.

Some IDE's (like VSCode) can show commit SHA changes between revisions. From CLI you can run git diff Cactus/Content (or your equivalent) to show the SHA change as well.

Diffing .uasset files

After a bit of use it became apparent that I couldn't diff .uasset files with the in-editor diff tool (using the Git version control plugin) as usual, since that plugin does not recognize our submodule as the repo and thus cannot detect any changes.

I didn't find a direct solution but I solved it by creating a python script that extracts the previous .uasset file from the submodule, and manually passes it to the UE binary with the -diff command. It actually works fairly well.

Link to script for reference.

Misc

It is a good idea to also add a .gitignore file to your Main repo. You can find a good ignore file on the GitHub gitignore repo which you should put next to your .uproject file (your equivalent of Cactus_Root/Cactus). Additionally you should add Content/* to that file so that you don't accidentally add content to the Main repo somehow.