The Great Journey

by Teh Lag (2015)

From TSC:E's original 2015 postmortem series.
(See also the 2022 followup: "There'll Be Another Time".)

---

I started modding Halo PC in early 2004. It is now mid 2015. I've been doing stuff with CMT in some form or other since March 2006, and I was in charge of the team from October 2011 until TSC:E's release this past March.

Rather than spending dozens of pages reminiscing about the old days when HMT 3.5's automatic model de-crapper was a big deal, I'd like to sign off with something that maybe has actual value. I think that there have been a lot of important lessons learned in those years about big personal projects. There have been plenty of successes, sure, but there have also been many failures — and I think it's a pretty well-established piece of wisdom that lessons learned from the former are a lot more meaningful in the context of lessons learned from the latter.

So, let's call this a retrospective on what sort of things a person and a team learns across 11+ years of project development.

Artwork by Sheik Wang for the Halo Graphic Novel

---

Lesson I: Control Your Scope, lest You Be Crushed under its Weight

It's dangerous to add new stuff mid-way through development. This one seems like it should really be a no-brainer, right?

Well, no.

Unfinished level geometry by Teh Lag for SPv2, the abandoned predecessor to TSC:E and SPv3

It happens all the time to mod teams and even professional developers, even outside of the video game industry. You get an exciting new idea and you drop everything to sketch it out and make it happen. You flex all of your talents and technical prowess. You work furiously and a week later you have it "working"! Just need to do a few more things and it's ready to go...

Except it's not really done, is it? Unless it was a really simple idea, you still have a lot to do before this thing is complete — "ready-to-ship" complete. This lesson is about recognizing the hidden (or maybe not-so-hidden) costs of unplanned additions to your project.

Unfinished level geometry by Teh Lag for SPv2

If it's HaloCE level geometry — yeah, we have it switching cleanly into and out of the stock level BSP, it's ready! But you still need to fix the view-culling portals because you didn't get them right the first time. You still need to fix the phantom collision geometry. You still need to place detail objects. You still need to paint the ground texture. You still need to place the AI in an intelligent manner. You still need to figure out if you have enough object slots in the scenario to even populate the damn thing. You still need to playtest and refine everything that you just added to be at the same level of quality as the rest of what you're working on.

Unfinished Sentinel Beam model by Teh Lag for SPv2

Unfinished specops SMG addon models by Teh Lag for SPv2

If it's a HaloCE weapon — you still need to come up with a set of VFX. You still need sounds. You need to make its gimmick feature stop crashing the game sometimes. You don't know if it's going to screw up the flow of a certain area. You don't know if you have enough tagspace in your map to support a new weapon.

Unfinished Spartan-III model by Doom and Teh Lag for SPv2

If it's a HaloCE character — you still need the textures. You still need to fix those couple of animations where it looks weird, or maybe you don't have the necessary animations for it at all. That active camouflage hack you talked about, we're still not sure if it really works against AI...

This extra work, these caveats, these details, these "yeah, it's done, except for that."

They will be what kills your project, wasting months (or maybe years) of you and your teammates' efforts.

The Creative Addiction

Go back to that first paragraph up there, where you dropped everything to work on this new idea. That's where we made our biggest mistakes in SPv2, and I believe that this is, more than anything else, what killed it.

You get addicted to going all-in on these new ideas. I, at least, know that I get a rush from it. It's the rush from watching something new come together, and the rush from having your peers praise you for doing something new and impressive. It takes over the way you work. You start creating things because it feels so good, not because you need them.

The problem is that once that rush runs out, you still have a lot of work to do to make your idea work right. You still, at the very least, need to maintain it and make sure it doesn't break between now and release. In the back of your mind you know this, you know it's going to be a total pain in the ass to properly unwrap and texture that hacked Sentinel Beam model. But now the rush is gone. So you just leave that extra stuff "for later."

Now you go back to working on boring stuff — stuff that's been around for a while, stuff that everyone knows needs to be done but nobody wants to do — but eventually you get another new idea and it happens all over again. Holy crap, wouldn't it be awesome to make a Flood Brute?! This vicious cycle continues, leaving a trail of half-implemented, half-working ideas in its wake. Once you have done enough to burn through that rush, you will stop. You will move on. And the ever-creeping scope will inhibit your ability to effectively work towards the project's completion.

Unfinished Flood Brute model by Teh Lag for SPv2

And here's the worst part: while your project begins to rot around you, you convince yourself that you're still making progress. You don't even realize it's happening. You think of things in terms of how much new stuff has been added, not how much time these new features have stolen from existing ones. Maybe you even make up excuses: "well, Dano isn't around to texture that model, so it's not like I could have been productive this week otherwise." You bullshit yourself and your teammates into thinking that you may as well be working on something, ignoring all the hidden details and cleanup work that that something will invariably bring with it.

Some time later, you have a project that nobody can finish. Nothing works the way it should, and there is no end in sight to the innumerable bugs and glitches and crappy placeholder art that stands between you and a dignified release. What's worse, there's now a team culture of creating new content to avoid having to fix up old content. Morale begins to buckle and break, and eventually, one day, you realize that it's all over.

It’s Okay to Say No

What I've learned after seeing this happen enough times is that it's not enough for something to be "working" to decide that you're going to commit to it. What you're really saying there is "it doesn't not work"; it's not a categorical failure. Even for the best content, the first draft is always just a first draft, invariably with rough edges and oddities and "well we can fix that before release." And, as you come closer to release, the rough edges stand out more. You eventually learn that there's a hell of a lot more work you need to do before this thing is "done" after all. Enough of this work will crush you. Keep things at arm's length until the creative rush dissipates and you can carefully evaluate the value and cost that they bring.

Unfinished spectacular level geometry work by BobTheGreatII for TSC:E, to which we had to say "no"

You need to sense when the creative rush is taking over and be able to say "no" to something new. Be willing to look a teammate in the eye and say "no" to that awesome thing they made, because you know that it's going to take another month to finish what they started and you won't have the resources to do it. Be willing to sit idle while your teammates catch up, so that you don't poison your project with added scope. (Chances are you'll think of something you already have that needs attention while you're idle anyway).

Say it takes a week to get some particular addition from the "idea" phase to the "doesn't not work" phase. As a rule of thumb, I've learned to estimate that between now and release, I will spend at least another 2-3 weeks bringing an idea of that scale to "release" quality. Then, realize that each prototype that takes a week to get up-and-running will steal 3-4 weeks total from your development effort. That quickly adds up to months, and eventually years.

Be willing to kill things if you see them becoming disproportionate time sinks. Stay focused on what works. Stay focused on the core of your product — and no, "more features" is not the core of your product. Don't sabotage your own team with reckless thinking! Real life can do a plenty good-enough job of that on its own.

During A50 2012's development, "details" like a textured Battle Rifle model and animations for the Brutes were unresolved until literal days before we shipped. The "detail" of texturing our new Pelican model never got resolved and we had to use the stock one. We came very close to not shipping on time, if at all.

Even during TSC:E, we struggled with this rule. We got burned — badly — with the directional lightmap system, because we were seduced by initial prototypes that looked promising. After publicly committing to the system, they ended up taking a good 3-4 months of dedicated work to finish. I am not sure if we made the right choice to stay with them.

TSC:E's DLMs turned into a quagmire after we had publicly committed to them too deeply to cut our losses

I've found that an effective way of managing this risk is putting things on a "maybe" list. Keep track of them. When you reach your next big milestone, review the list and see if there's anything that can be quickly taken care of. Additionally, if someone is passing through a related part of the project for an existing feature, take the opportunity to get a sense of the addition's feasibility. If you think that you have the necessary resources, go for it. Otherwise, put it back on the list for next time. Keep momentum going on things that you've already committed to, while allowing an outlet for low-risk additions.

These are hard calls to make. Sometimes a risky move pays off. The alternate-route cave segment of TSC:E didn't come in until the very end of development, but we thought it was worth the potential cost. Sometimes a risky move does not pay off. We had to abandon a lot of high-res structure modeling in TSC:E after we realized we couldn't make it fit under the game's data limits.

You won't be able to do everything cool that you want to - that's just a fact. It is why game developers stay around to work on sequels.

He was a smart man, that Yoda.

He says it best in Empire. "All his life has he looked away, to the future, to the horizon. Never his mind on where he was. What he was doing."

---

Lesson II: Iteration and Feedback Are Your Keys to Victory

(For this section in particular, I want to give special acknowledgement to a certain Software Engineering professor that I had mid-way through production. His teachings continue to resonate with me, and had a profound effect on the way I thought about -- and continue to think about -- this project and software development as a whole.) (Ed. 2022: this is perhaps less true now, but it was 2015 and I said what I said.)

"But, Teh Lag, if we never add new feature to our mod, we're going to end up with a lame-ass mod!"

This is where self-discipline regarding scope creep pays off. All those details — those things that you might not get to at first — you'll be able to nail them down. You'll be able to polish them to a fine sheen, so that your product becomes more than just some crap some people threw together. It will become a product.

You'll have time and resources to catch your mistakes so you can fix them before they do too much damage.

The Value of Changeable Designs

This is something that we started learning during A50 2012.

We all wanted something that was more than just some crap some people threw together. We knew that things didn't work with SPv2, but we needed a way to make sure they didn't happen again. We didn't have a good sense of direction. We didn't know the specifics of what we wanted.

The only thing that seemed to work was to try something and back out quickly if it wasn't working. Additionally, we often had to make several passes at systems before they solidified. (Incremental development is, incidentally, a fundamental tenet of the Agile software movement.)

We were lucky enough that TSC:E's Beach LZ didn't have to change much

The original Security Station was very boring — we had to re-design almost the entire thing

The re-designed Security Station became many peoples' favorite section of the map

The Security Station was an "a-ha!" moment. We were lucky enough that we hadn't invested too much into the area before we took a second look at it. The re-worked design was a huge success, and it taught us the real value of being able to revisit things — even things as major as a core level segment. For basically every area of the map after that, we were very conscious of the fact that we might need to do some radical changes before the area was satisfactory.

We learned to create minimally-viable prototypes because we needed to minimize the cost of future improvements. Resources were scarce, meaning that once we committed down the expensive path of final polish for a feature or gameplay area, there was no going back. As a result, most areas were implemented in the most bare-bones way possible, with care taken in the source files to accommodate potentially large future layout changes. (No fancy Forerunner strips or bevels until you're sure that the layout is what it should be).

A minimally-viable Map Room

An even-less-minimally-viable Security Bridge

We changed our minds so many times on the layout of the Security Bridge that we stopped making BSP changes — we didn't have the time or resources. For over a year, we went back and forth using ingame scenery objects until the layout was just right. By the time we were finally able to prepare its BSP for the final release, we had figured out exactly what we wanted for it.

The primary instances of things objectively not working, even in TSC:E, can be traced to people investing too heavily in a big-design-up-front solution to a problem, and thus finding themselves unable to fix issues in their big design. For example:

• The TSC:E map room was an over-designed over-compensation for the lack of interest in other interior combat sections. We had too little time to re-work its massive scale before having to move to the final polish phase. It had to ship with portal errors.
• The security cave was just barely beaten into submission before release, after a gargantuan effort to make its portals and BSP switches behave correctly. It also had to ship with portal errors.
• The latter half of the second Warthog run was concocted at the last minute because that section of the map wasn't playable until the last year of development, and thus we got no feedback on its layout until it was (nearly) too late. We ended up having to scrap half of it. The awkward jumps down to the Shaft Lid and again down to the Cartographer Field are the legacy of its rushed final implementation.

The Value of Feedback

This is the "feedback" part. It's not just about people telling you "yeah the mod is cool". It's about maintaining constant lines of communication among people who have involvement in the project. It's about having the freedom to respond to feedback — even radical, destructive feedback — without getting bogged down in 10,000 other features that you still need to finish too. Trust me: the time we could have spent adding another route from the end of the Cave to the LZ was better spent improving the layout of the second Warthog run.

During the security station redesign, the environment modeler (Siliconmaster) and the encounter designer (me) were going back and forth regularly on how the map should be designed, and it's what allowed us to come up with the Security Pit idea. And so, before we even got that first prototype of the redone security station ingame, it had already been through over a dozen feedback-iteration loops.

After the security pit, we moved on to the Hunter Elevator complex. This time, before even starting to model a game-ready mesh, we created a 3d sketch prototype. From there, we started thinking about how combat would flow. The core idea — "an Elevator that Hunters can chase you with" — was expanded and refined until we were comfortable moving ahead with an ingame version.

Rough 3D sketches like this became common as TSC:E's development continued. Further iteration on the area trimmed some excess space and removed a "boxing ring" visual metaphor that was originally going to surround the elevator. Metal debris serving as cover in the Hunter complex was implemented as scenery; this allowed us to back out of it when it became clear we wouldn't have time to pull it off convincingly in the BSP. (It was only at the very end of development that we decided to back out).

The initial 3D sketch

The refined 3D sketch

The first in-game draft

A near-final draft; metal debris cover still present

The Value of Process

And so, here is where we arrived: If you don't create content for your project with the intent that some day you'll probably come back and change something to make it better, you're creating a trap for your future self. There's nothing wrong with having to go back and change stuff. There's only something wrong if you're unable to make the change, and you have to ship with something sub-par.

You don't need to magically get stuff right the first time. You just need to have mechanisms in place to realize when something isn't working, and then quickly move to address the problem. This helped breed a healthy "do more with less" attitude among the Evolved team, as we came up with inventive ways to add flavoring and character to areas that were often very minimalist in implementation. It gave us a framework to continually improve the mod's value without shoveling on more content.

This is what separates a product that's intended to be good from a product that's good by accident. This is process, I think. It's the bigger picture of how you and your teammates get things done. It's the infrastructure and the self-discipline to realize the things you want to make.

The next section is a good example of that. Maybe.

---

Lesson III: Take Collaboration Seriously, Or: Use Source Control

If you're coming from a background in software development, you know this already. Feel free to move on.

If you're not — do a search right now for version control software and run through a tutorial for one. Git, Mercurial, whatever. Decide on one that you like. It will change the way you work.

CMT used Mercurial during TSC:E's development. But it wasn't always that way.

From when I joined in 2006 through September 2011, CMT members collaborated by sending each other archive files (.zip, .rar, .7z) containing various data files. It was the sender's responsibility to make sure they sent out all relevant changes, and it was the recipient's (usually Masterz or myself) responsibility to correctly integrate the new content. Usually files were sent through AIM.

This was a nightmare, and we didn't realize it until it was too late. Files got lost. People forgot to send needed changes. People forgot to overwrite their own changes. I sent out a copy of the SPv2 Sentinel Beam files no fewer than 3 separate times, and they never made it into an official build. Additionally, it was impossible to propagate changes back to the other members, unless they wanted to manually extract content from a test map. Peoples' content got out of sync and began to rot.

This inability to effectively collaborate among a large team was another one of the key factors in SPv2 collapsing under its own weight. Nobody could do quality control on anything. They couldn't even know if their content was correctly integrated.

Did I ever send out that updated version of the A50 BSPs? I will never know.

Iteration 1 - Synced Files

Fast-forward to October 2011. CMT had been resurrected to work on SPv3. We were going to do things and do them right. After a month or two of exploratory work, it was decided that I was best-suited to lead the development effort. Having reflected on the nightmare of SPv2-era collaboration, we came up with a new solution.

I rented some dirt-cheap storage space on Amazon's S3 storage service, and set up some file-synchronization software for myself, Ifafudafi and Masterz. We went through a painful week where everyone uploaded their tags\cmt\ directory and smoothed out the differences among the versions. Then that week was over.

For the first time, we were actually able to work on things and know that the others would get our work. More importantly: we were able to know what we had changed when we synchronized our files out to the server, and also what others had changed when we synchronized the server's files to our own directory.

And, even more importantly, we started flushing out flaws in the CMT content base. Invisible dependencies on modded versions of stock game files. Dependencies on files outside the \cmt\ directory. Outdated and incorrect properties that everyone could see and call out. At long last, anyone could build a map if they wanted to, at any time, just by downloading the latest content. We started doing semi-weekly test builds for the rest of SPv3A50's development cycle.

It wasn't quite version control, but the results were mostly good. I think.

We remained on this flat, un-versioned storage system for two years.

As time went on, new problems began to come up. People would have files with bad timestamps, causing needless sync conflicts. People would overwrite files without thinking, and then have to ask another member to sync in a previous good version. People would have colossal sets of changes and tracking what was and wasn't supposed to be uploaded was miserable. And, outside of the three team leaders, nobody was directly uploading files because managing and avoiding incorrect changes was still a nightmare. Adding more people to the mix would have made it impossible.

Iteration 2 - Mercurial

Fast-forward to December 2013. The flat sync system had become basically non-functional.

I had long ago been warned against mixing binary files with source control systems, and the HEK's tags are nothing but binary files. But, it was clear that we needed something new. And I had already been eyeing a system to manage our script source files — text is what source control software was designed for.

We also needed a way to reliably allow the other team members to integrate their own content — we had a lot of work to do and some recent real-life occurrences had made it clear that we wouldn't always have team members around who knew how to work with the entirety of the game's content pipeline.

We set up a Mercurial server on a rented Amazon EC2 Micro instance. (Similarly dirt-cheap, though not quite as cheap as S3). Ifafudafi and I went back and forth on some conventions for a few weeks, and then we opened it up to the rest of the team.

This screen was the most beautiful thing about developing the mod. Probably.

The difference was unreal. Every set of changes, marked with a timestamp and an author and a description and a list of affected files. Every version of every file, listed and available for inspection at any time. Most importantly: we could iterate on content safely and rapidly, with all of our iteration history logged and organized for us. We started making dozens of commits per week — even per day — because we could finally isolate units of work safely and clearly. To be sure, we hit some snags whenever we had to merge changes on a binary file, but permission locks on the main development branches made sure that project leads were able to resolve them in a controlled manner.

We stayed with this system for the remainder of TSC:E's development. I would call it a success. I would never start another development project — even a game development project with icky binary files — without a version control system guarding my work.

Even so, there were failures here. The highly technical nature of the system was daunting for many members, which was a challenge for the duration of the project. Additionally, the gargantuan CMT content base began to place strain on the infrastructure hosting the data. People uploading and downloading changes often had to try multiple times before the request would go through; Mercurial and the poor little EC2 Micro instance weren't designed to handle hundreds of MB of data coming in at once.

Iterate One More Time

If you ever stop trying to improve your process, you're doomed. If you don't know how to improve your process, go out and do some reading until you find something that looks applicable.

If we were to iterate again on this...?

I would split off the core tagset from the large BSP files. BSPs were big. Bigger than the system could handle. I might move them onto a separate repository, where only the people working on the scenarios have to deal with giant changesets.

We did not apply these tools to ensure that raw data — .max files, .psd files, and the like — were kept in sync. Each artist had wildly different preferences, and we never agreed on more than consistent naming and relative image paths for shared .max files. This was probably a mistake. There are now many assets on the CMT content base for which nobody has well-organized source files. Tracking changes to art files would be a top priority. Maybe the solution to this would be related to the solution for BSP files.

And, most importantly, I'd try to get more team members familiar with the system. The prolonged crunch to get TSC:E out the door meant that we didn't have a chance to all sit down and get to know the system. Some people never really got the hang of it. As the team leader, I view that as a failure on my part.

---

Lesson IV: Know and Respect Your Team and its Limitations, or: "If you want it done right..."

The majority of TSC:E's development was handled by a core team that did not exceed 4 people. For a period of time, LlamaJuice was a 5th. Despite the large content base CMT had, it was a very resource-constrained environment relative to the scope of the project. People's availability to complete tasks was not guaranteed: all involved parties either had obligations to school or a job or both.

Every feature and content addition and even every bugfix had to be weighed against how much it would set things back, and whether a sufficiently-competent team member would be able to complete it in a reasonable timeframe. (The Shredder came very close to being cut, or at the very least reduced back to a red clone of the Plasma Pistol). Working in an environment like this, we all had to learn to be nimble and self-sufficient. If we didn't, the project would not have finished. I ended up modeling a large portion of the Hunter's geometry in late October. Ifafudafi and Silicon had to set up much of the interior of the level while I was gone for real-life reasons. I could go on and on about this, but this article is getting embarrassingly long so I'll stop it here.

Here's what we all learned about teamwork. Some parts of this lesson were harder to learn than others.

1. Consider whether your efforts are going towards attainable goals, and whether you and your team will be happy with the time investments you've made a year from now. Prioritize accordingly. Don't sell the team on something that you can't know that you will finish — then you will have to deal not only with the wasted effort of partially doing that thing, but also the cost of backing out the investments everyone else has made under the assumption that that thing will be finished.
   
2. Consider the stake that you have in the project's success and whether you would be willing to take charge of problems outside your current domain to ensure that the rest of your work makes it to a shipped product. Multiple members need to take a hands-on interest in all aspects of the mod, not just their portion of the 3d art / sound / hud / scripts etc.
3. If you bury your head in the sand and just throw your content over the wall hoping that things turn out okay, things will not turn out okay.
4. If an unfinished task is not dedicated to fixing problems in existing content, prepare to do it yourself or assume it will not be done.
5. If you find an issue and you - not somebody else, you - don't take action to ensure it gets fixed, assume it will not be fixed.

Avoid depending on work that you're not willing to do on your own.

Demon Trees and God Rays — last minute additions that only worked because they were appropriately prioritized

A lot of the cool things — a lot of the working things — in TSC:E's end product only work because someone took direct responsibility for them. Build trust in your team and do not fear territoriality. Prove that you can do the job right and everything will be okay.

---

Closing Thoughts

I guess what this is all building towards is some big list of Things to do if you're working on a big fun project, and you have limited resources with which to make it happen.

I don't have a comprehensive list, but I can give you these:

1. I have been taught that a great leader doesn't say "go over there"; a great leader says "follow me". If you're just an ideas guy and you can't get that first working prototype of something off the ground, your project is already doomed.
2. Think about what you want to get out of the experience and what will give you the best chance of realizing your goals. Then, keep those goals at the forefront of everything you do. If you need to change goals mid-project, keep in mind that some of your prior assumptions and planning will be invalidated.
3. Consider whether new features are appealing because they represent the opportunity to avoid doing serious cleanup on ones that already exist. Do not attempt to put out a fire with gasoline. Marketing-driven design and scope creep are poison for independent projects; they are good at producing "wow!" content for the public to ogle but they seem to rarely correlate with the discipline and resources needed to ship the things that were promised.
4. Strong personal investment by contributors is, in my estimation, the most important factor in fan projects' success. If you're the person in charge, it's your duty to create an environment that fosters that investment. Give people autonomy, but be there to back them up when they need help. Make sure that everyone can understand their work in the context of the whole — this means regular internal testing, and ideally a build system that everyone can use. Make sure that everyone feels like their say in the end result is proportional to the amount of work they put in. If the people with the most control over the project are contributing the least, there's probably an issue.
   
5. Strong direction is, in my estimation, the next-most important factor. It allows people to establish a framework for filling in for each other while retaining project coherence. TSC:E wouldn't have been finished if Ifafudafi, Silicon and I weren't able to branch out and handle things that used to be only one person's responsibility. If you're not sure what your direction is, start iterating on things fast so you can find it. You don't want to find out what it is 2 years later, after you've reached the end of your collective motivational limits.

Give a damn. Make something great.