March 1st, 2015

The Case Against Data Driven Design

Data driven design is a hot topic among the industry. But what does it mean? I couldn’t find a good definition of it on the web, so I’ll give you my definition. Data driven design is where most or all the elements of design are out of the code and exposed to the designers. The main difference in my mind is the division of labor. Data driven design has programmers doing less and leaving it up to designers/artists to implement. Having programmed gameplay for games both with and without data driven systems, I would say that data driven design is actually a worse way to create games even though many people feel that it’s the “right” way. In looking for a good definition, I stumbled on other blog posts about data driven design. On Kyle Wilson’s Blog, he says that the game engine shouldn’t contain a single line of game code and should be completely data driven. On Dan Cook’s blog, I found an interesting post about data driven development. I’m not sure he has the same definition of data driven design as I do, but he asserts that it’s a good way to keep down costs, but you can’t have any real gameplay innovations because that would be too risky. I would argue that using data driven design, you could innovate gameplay by giving the designers more control over the behaviors of in game characters possibly by using a scripting language. But, let me explain why I think data driven design is a bad idea.

Problems with Data Driven Design
Data driven design is about exposing as many aspects of development to the designers (and to some extent, the artists) to lessen the burden on programmers and to reduce the back and forth collaboration between programmers and the rest of the team. Control can be exposed in many ways. It can be exposed through scripting languages. Common choices for scripting languages are Lua, Ruby, Python or a custom language. There are several problems with scripting languages. They can be slow since they are ordinarily interpreted. Custom hooks need to go into the engine to expose the functionality that should be scriptable. If the scripting language is compiled, a compiler has to be implemented, which isn’t that difficult for most software developers, and I’m sure there are many open source projects available for common scripting languages, but some form of a debugger needs to be implemented, which is going to be a lot more difficult to find if you are doing console development because it’s on a unique architecture and needs to be debugged remotely. If you don’t have a debugger, debugging assembly generated by a compiler is going to be very difficult, and there’s always a lot of gameplay bugs, so this slows down development considerably. So, I imagine most game developers that use scripting languages choose to interpret them. Interpreting solves some of these problems, but debugging an interpreted scripting language is also tricky if you don’t have a debugger for it. You can step through the interpretation engine, but it’s tricky to see what is going on since you are seeing the execution of the interpreter rather than the actual code. Some interpreters that are optimized are very difficult to debug. The worst problem is when there’s a bug in your compiler or interpreter.

On top of this, designers are coding. Designers usually don’t have a background in computer science, and although scripting languages are pretty easy, they aren’t trivial to learn, nor will a good designer necessarily be good at them. Imagine a situation where you hire someone and find out he is great at level designing, but can’t write scripts to save his life. Do you lay him off? Do you force one of your other level designers to do scripting? Maybe they aren’t very good at it either. Now you have someone who is very good at “design” at your studio, but is terrible at scripting, so it may be hard to find a position for them. Oftentimes, people who don’t have a lot of experience programming can make a lot of mistakes. Sure they’ll get better with more experience, but they might not be cut out for coding. On top of this, they usually aren’t equipped with either the tools or the know how to fix their own bugs, so the regular programmers are forced to do it, and the bugs can be a real pain for programmers to find as I mentioned in the previous paragraph.

Many studios don’t give the designers scripting languages. They create an editor that is either using plug ins for a 3D modeling package or create their own stand alone application. The designers use this to create links between widgets that determine the behavior of the objects in game. They may have some widgets to do logical operations, counters, conditionals, etc in order to design complex behavior. The widgets themselves may have a varying amount of parameters that the designers can control. I think this is still a bad idea. For one thing, using a lot of widgets to essentially form a scripting language doesn’t work out too well either. Let’s say when the player slays the Fearless Three and a Half Headed Sea Dragon of Krandell you want to award him the Dongle of Incontinency Prevention if he/she hasn’t already bought it at the store. Otherwise, you want to award the Armulet of Ambivalence. Although it sounds easy, this can end up being very complicated. When the dragon sends itself the destroyed event, designers may send the player an event querying whether or not the player has the dongle. They could pass another widget as a parameter, so the player would send the false event to that widget. That widget can have a link to send the dongle a give to player event when it gets the false event. That same widget can have a link to send the armulet the give to player event when it gets the true event. Now, what happens if the player uses a temp save option and turns off the console? We’d want to store that the player has the dongle. Now what happens if the player turns the game on dies before hitting the checkpoint? The player will have to fight the dragon again, so they shouldn’t have the dongle. How would a designer express this with links and game widgets? A problem such as inventory would most likely be solved in code with an inventory system, but regardless of how many systems are in place, there are always situations where it is tricky for a designer to handle all the special cases that arise. Because of this, bugs can occur, and it is difficult for either a designer or a programmer to solve it. Designers don’t normally have the tools so they can see the logic that they have created in the game, and it’s difficult for programmers to follow a complicated web of links in order to track down where it’s actually going wrong.

Even though many game studios create widgets that designers can control, on short development titles, the widgets usually require quite a bit of tuning and tweaking. As a game widget evolves and improves over the development cycle, it will have new or modified parameters. If designers are setting this up, the parameters must be exposed to them, so some sort of documentation needs to pass the information along. Maintaining a wiki or some other documentation about how things work can be tricky if you don’t have a full time technical writer on staff. Programmers aren’t really known for their writing skills, so your descriptions in the wiki may not be clear. The worst thing that can happen is not when a designer doesn’t understand it, but rather when a designer thinks he/she understands it and builds off those false assumptions. This can turn out poorly because programmers will be spending their time documenting (and not be very happy about it), and designers may not even get the information that they are trying to communicate.

But Sony Santa Monica Did It!
Now some readers may have read gamasutra’s excellent coverage on Tim Moss’s talk about the development of God of War. It describes the very data driven approach that the engineers took on God of War. You may be thinking that God of War was a very successful game, and they used data driven design, so why shouldn’t we? What people may forget when they look at God of War was that it took 3 years to develop. Even though it had very few programmers, it had a very large development team for a PS2 game. I’ve heard rumors that there was a lot of turnover on the project because it was very difficult to work on. Maybe data driven design ends up being more efficient when you are developing a game for that amount of time, but few titles are afforded the luxury of a three or more year development cycle. In my experience, companies want to get a game out every year assuming they already have technology for the platform (which Sony of Santa Monica did since they released Kinetica, but they may have started from scratch). Obviously, the transition to new consoles (and new technology) is always difficult and time consuming, but God of War didn’t have to overcome that hurtle.

If Not Data Driven Design, then What?
If you have 3 years to make a game, then maybe data driven design is a good way to develop your title. If you don’t, the people who are going to be the best at coding your gameplay logic will be a programmer. Minigames, AI, player control, and any other things that change from game to game are best left in the capable hands of programmers. The people who are best at designing effects for real time rendering are also programmers. Unless you have a very technically savvy artist, a creative programmer will be able to create an effect that looks as good and gets better run-time performance, which is crucial to have a good looking game. Now, I’m not saying that artists can’t design particle effects, but usually the actual implementation should be left up to a programmer unless you have a technical artist. You also need to hire talented gameplay programmers that have a real feel for design. Ideally, the gameplay programmers are given a rough sketch of the design then free reign to create something great from it, and feedback after they are finished. Because the programmers themselves are able to tune and tweak either the effects or gameplay, it is much faster than the constant back and forth between the departments. Rarely will programmers be twiddling their thumbs waiting for someone to get out of a meeting before they know if something is the right direction or not. Gameplay programmers are empowered to pick the direction that they feel is right, and go down it. If it doesn’t work out, then the designer and programmer can work out a new direction to try. Gameplay is very difficult to specify. No designer is good enough to write a spec that is both detailed enough to include everything and is guaranteed to be fun. There are always a ton of unknowns in design – many decisions can only be made when the game is playable. In fact, that’s why it’s often bad to overspecify design at the beginning. At iterative approach is usually much more successful. Oftentimes, designers try to write a document with all the details upfront only to find out that there are some core aspects that need changing. It’s much better to start with a general idea, see what’s fun and then refine the design from there. If a company has talented gameplay programmers, they can help flesh out and improve the design with very little feedback from designers. This saves time for both the designers and gameplay programmers, which helps speed up the development cycle.

As team sizes increase, the more communication dependencies become a problem with development times. Fred Brooks’ seminal work on software development, The Mythical Man-Month, discusses the negative ramifications communication can have on a project’s development time. If the person in charge of implementing it (in this game the gameplay programmer) is the driving force behind making that feature fun, then iteration time is greatly reduced because there is little communication overhead. This is somewhat similar to the surgical team that Harlan Mills proposed for software development, except with even less overhead. The gameplay programmer is the surgeon – the driving force for implementing the feature. The designers are more like the hospital directors – they give feedback and direct the surgeon AFTER the surgeon completes the surgery. This means the gameplay programmer has free reign to make the feature fun, and only shows it to the design department for feedback after he/she is satisfied with the results. Too often when gameplay programmers take liberties with the design, designers can feel threatened. Game development should be collaborative and with talented gameplay programmers, they will know how to tune and tweak a game.

Although I talked at great length about the problems with data driven design, there are also problems with this development methodology. However, I think these problems are far outweighed by the benefit of having things programmer driven rather than data driven. Good gameplay programmers are hard to find especially now with team sizes increasing because of next gen. I agree, but I would argue that good designers are hard to find also. A studio that has less designers and more good gameplay programmers will have a higher productivity for the reasons I specified above. Another problem might be that programmers typically cost more than a designer with similar experience. That’s also true, but if it speeds up development, then it’s probably worth it. Slower development is expensive especially if you miss your release date like EA did with the Superman Returns game. Instead of coming out along side the movie, the game will be released at the same time as the DVD. I can’t guess exactly how much that cost EA, but I have a feeling they would have made a lot more money if the game were out alongside the movie – not to mention that the game would have been cheaper to make because it would have been in development for less time.

To sum it all up, designers should do just that: design. Let them design, and let the programmers implement. It takes programmers with a good gameplay sense, and yes, they may be hard to find, but the results are that you can develop a game faster and probably less expensively too. If you have a very long development cycle creating a lot of tools and making it data driven may pay off, but most games are on a one to two year development cycle, so I think in most cases, data driven design is a bad approach to game development.

12 Responses to 'The Case Against Data Driven Design'

  1. 1Joseph
    September 15th, 2006 at 10:15 am

    Actually, Superman had an extremely ambitious development time (shorter than an iterative game like Madden). From talking with people, its problem was more of a managerial one than anything.

  2. 2Matt
    September 15th, 2006 at 11:01 am

    Thanks for the feedback about the Superman Returns game. I’m sure it had a very ambitious schedule, but I was just trying to illustrate that slipping a release date can cost quite a bit of money for the company. I didn’t mean to imply anything about the development of that particular game.

  3. 3Aevaans Simba
    February 27th, 2007 at 7:52 am

    That was really interesting because i have been struggling to find jthe same defination

  4. 4Dan
    May 13th, 2007 at 6:47 am

    I would think data driven design would be mored focused on data – like columns of stats and various switches, rather than fully blown scripting code such a Lua and Ruby etc.

    I guess a lot of these terms are quite new and need some time to be bandied about a bit before they’re given solid definitions.

  5. 5Tim
    February 6th, 2009 at 6:47 pm

    Actually… Unreal engine only exposes single variables or lists of variables. Similarly, variables can be exposed through serial/hierarchical listing with languages like XML.

    The problem comes down to the fine line that everyone keeps forgetting: Designers aren’t programmers. Don’t treat them like one – don’t give them a ‘development environment’ with a scripting language, and variables, and maybe even a debugger.

    Programmers end up sitting there writing script hooks and designers sit their writing crap code that fucks everything up, then programmers have to come fix it! I always wonder why people do it to themselves?

    Their games deserve better.

  6. 6Matt
    February 17th, 2009 at 9:54 pm

    Tim: I really wonder how much of the development of the gameplay code at Epic is done by programmers and how much is done by designers. I know at some other studios, their programmers code in UnrealScript. Some programmers prefer the use of a scripting language and think it’s faster. Personally, I would much rather use C++, unless by some miracle, the scripting language generated really optimal code and had an amazing debugger.

  7. 7Amumu
    September 16th, 2011 at 11:29 am

    Hi Matt,

    I posted a question related to your article on Stack Exchange:

    Please visit if you have time. I want to hear your opinion as well.

  8. 8Amumu
    September 16th, 2011 at 10:45 pm

    Hi Matt,

    That was a great answer. There is something I want to note.

    “You can share game code between games regardless of what method you choose. I’m not really sure what you are getting at with this point. Even if you aren’t using a scripting language or visual tool to define some behaviors, you should be architecting your gameplay code into reusable components as much as you can”
    You are right. In the programmer’s perspective, either code or visual component is reusable. However, as you pointed out, the community might need it too. Also, I’ve seen a lot of games, the next sequel is just a difference in gameplay/graphics. I think by using visual editor will greatly enhance the productivity (also for other applications which are not games, I prefer to write code than drag and drop like in Visual Studio, since it is more controllable and faster), because we don’t have to look at the old codebase and manually take out what we need.

    I can imagine the game engine is like the IDE for the game, pretty much like Visual Studio to the general application, where things can be dragged and dropped. Except Visual Studio is IDE for general application, while game engine is for game specific. What do you think about this?

  9. 9Matt
    September 17th, 2011 at 11:53 am

    I’m a little lost. What community are you talking about?

    Also, I’m not sure you have the right idea for starting a sequel. I’m pretty sure everyone always starts with the game code for the previous game. You don’t have to look at the old codebase because everything there is still there. If you decide that you want to get rid of something, you can delete it or change it.

    If you are working with a scripting language or some sort of visual scripting language, you have no advantage in this situation.

    I’m not sure your analogies are helpful in this situation. Game logic is complex interdependent systems. Obviously you should make everything as independent as possible and as reusable as possible, but I don’t think anyone has ever architected a system where you can just drop in components independent of each other and call it a new game. If such a system existed, it would be pretty horrible to work with because you couldn’t make any assumptions about what other systems are there and what sort of objects your characters are interacting with.

  10. 10Amumu
    September 17th, 2011 at 9:59 pm

    Well the community I mentioned is the community of that game. Take Warcraft for example. With the World Editor, game rules and game scenarios can be created completely, just like the original Warcraft (which I don’t think not many people play now) and DOTA (which is still very popular). Original Warcraft, DOTA and various game maps were created from that program, which brings Warcraft lots of popularity. Starcraft II also has such map editor, but more powerful, that people can create role playing, action game or FPS game from it:

    With this approach, users don’t have to look at the codebase, but they still can reuse the game elements. But of course, the editor is more complex than just drag and drop.

  11. 11Matt
    September 18th, 2011 at 9:14 am

    Supporting mods may or may not make sense for a particular game. A lot of games are console only, so mod support is not really a concern. A lot of games with mod support such as id and Valve’s games require you to code most of your game logic rather than opting for visual editors.

    Warcraft 3 took over 4 years. Starcraft took 7. If you can ship a product when it’s done, you can do these types of things. If you are building your own technology, and you don’t have a ton of time and money, it’s not possible to get tools at Blizzard’s level.

    When making a game, your biggest concern should be finishing that game. If you can’t put together a great game, then it doesn’t matter how good the mod support is. It won’t catch on and people won’t make mods. When you have a great product bringing in a lot of money, then you can spend a ton of time making a sequel with all the features you want.

  12. 12Amumu
    September 18th, 2011 at 10:22 pm

    Thanks. Things make sense now.

Leave a Response

Everything on Binary Creativity is © 2006-2010 Matt Gilgenbach. All rights reserved. | RSS | Comments RSS