Naughty Dog Audio Team

If you were born before the 90s, I’m hoping that you know about Naughty Dog. After all, their Crash Bandicoot game franchise has sold over 40 million combined copies, their Jak & Daxter series has sold over 12 million copies, and their most recent game, The Last Of Us, which topped everyone’s “Best Of 2013″ list, has sold over 6 million to date and is being turned into a movie.  It’s fair to say that they haven’t made an unsuccessful title yet.  Given the recent acclaim for The Last Of Us, I as an audiophile felt the need to talk to some members of their team in Santa Monica about what it was like to handle the audio duties for the project. So I had a round-table chat with Jonathan Lanier, Phillip Kovats, James Barker, Erick OCampo , Derrick Espino and Scott Hanau about both Naughty Dog and The Last Of Us.

Thanks a lot for taking the time out of your schedules to speak with me. Could each of you tell me about your positions at Naughty Dog?

Derrick : I’m a Senior Sound Designer who works for Sony Computer Entertainment America (SCEA), though I’m currently located at Naughty Dog.

James Barker: I’m the Dialogue Supervisor at Naughty Dog.

Phillip Kovats: I’m the Audio Lead for The Last Of Us game.

Erick Ocampo: I’m also with SCEA, and am located at Naughty Dog as a Sound Designer.

Jonathan Lanier: I’m the Audio Programmer at Naughty Dog.

Scott Hanau: I work for the Music Department of Sony’s Service Groups, which services most of Sony’s first-party games. For The Last Of Us, I was the associate music supervisor who scripted the music into the game, and worked with Neil Druckmann to decide where the music plays. We also produced 90′s minutes worth of in-house music for the game’s soundtrack CD.

What kind of interplay exists between the different people on Naughty Dog’s audio team? Can you tell me abut how you guys work together and how the corporate structure comes into play with that?

Jonathan: Phil, James and myself are employed 100% by Naughty Dog, whilst Erick and Derrick are employees of SCEA. Naughty Dog is a wholly owned subsidiary of SCEA, so we have a direct relationship with them in a corporate sense. Sony has an internal Product Development (PD) staff, who are loaned out to different projects as the need arises. So when Naughty Dog is in-between projects, people like Derrick and Erick move on to work on other Sony projects, and then when we need them they come back here.

Phil: We’re very much a team here at Naughty Dog and Sony. For example, at the end of the project, Bruce Swanson, our Audio Director, got involved  to help Derrick and Erick with the opening level of the game. We also had another ten people from Sony’s PD Sound teams in Santa Monica, San Diego and Foster City help out. So we leaned heavily on them when we need help.

Jonathan: But guys like Derrick and Erick aren’t just contractors that we use to just fill space. Even though they’re not Naughty Dog employees, we kind of consider them honorary Naughty Dog staff. For example, Derrick has been here since 2009, which is a while.

Scott: I’ve known Phil for a long time beforeThe Last Of Us. We worked on God of War 2, which was before either of us worked with Naughty Dog. So thankfully, there’s no ego that gets in our way, and it’s the same with other team members. We all know how to recognize good ideas and know what the deadlines are for our projects, so we don’t have to beat each other up over things like that.

Would you say that there is a hierarchy at Naughty Dog, where staff members play specific roles, whilst answering to their supervisors? Or are you guys allowed free reign to be as creative as you want?

Phil: There is a loose hierarchy. Naughty Dog is a pretty flat company, and we don’t have game producers per se. We have Evan Wells and Christophe Balestra, who are the co-Presidents. Underneath them are the creative, art and game directors. Following that, the studio has Leads for every disciplines, such as animation, cinematics, audio, environment art. Under the Leads, there are teams of people that we trust and work with to get things done.

As a Lead, I don’t really sit around and say “You do this and he does that“. For The Last Of Us, I needed to be able to delegate duties to people I could trust. So Derrick was my Senior Sound Designer. Him and I would work on the sound effects in the game. James worked with me on the dialogue. Scott worked on the music part, and Jonathan handled the technology aspect. We naturally let things flow based on what the project was calling for at the time. Derrick and I worked on the Infected sounds early on, and then when Erick was brought on later, he took over that and helped create the rest of the sounds for the Stalkers and Bloaters.

As individual team members what was the highlight of making The Last Of Us for each of you?

Derrick: Going into it, we knew that this would be a rare opportunity for the audio team, because The Last Of Us was designed to need so much attention to sound. Given the use of not only sound, but also silence, the game had to have a pretty strong sonic signature. That was a bit scary, but it was also exciting, because it’s common for sound designers to have to struggle to get their sounds heard in the final mix, but that wasn’t the case here. Consequently, the sound quality had to be very good. So we focused on everything, from characters fighting to dust falling off a door.

The ultimate highlight for me was inventing the Clicker sounds with Phil and Neil Druckmann. Once it finally came together, we knew it would be powerful.

James: Since my job was to oversee all the dialogue, I got to be a part of the game from the very beginning, during all of the Motion Capture and Automated Dialogue Replacement (ADR) sessions, when all of the story content was being created by our actors. I also worked with a lot of programmers, who wanted very intensive dialogue systems for the Artificial Intelligence (AI), and we fleshed that out together. We also built breathing systems for all the characters and tried to make it as realistic as possible. It was nice to see it all come together.

Phil: I was here almost from the beginning, when the initial pitch was given by Bruce and Neil, and got to watch the whole process go from inception to completion. My favorite part to work on was the mix of the game. One of the things we were mindful of was the inspiration that Bruce and Neil took from the music in movies like “No Country For Old Men” and “The Road“, and other simplistic scores that use things like negative space as part of the audio experience. Usually, games have a lot of sonic information, like explosions, gun-fire, magic, etc. So to create something which was so unexpected from that point of view was different.

Erick: I was the last one to come on to this project, and the team had already spoken with me about working on the Infected. The biggest highlight for me was getting the rest of the Infected to sound cohesive as a family of creatures. I also had to create convincing sounds of evolution for these creatures, as they morphed from Runners to Stalkers and Bloaters. They had to play back intelligently, rather than just fire off sound effects left and right.

Jonathan: I’m sort of the odd one out here, because I don’t do creative sound design like these guys. I’m a programmer, but Naughty Dog has allowed me to pursue my passion in both programming and audio. Many companies don’t even have the role of an Audio Programmer, but we like being set up that way since it allows me to provide my team with technological assistance they need to make the vision happen in the game. With the exception of the dialogue manager, which was written by Jason Gregory, most of the other audio-related stuff  was done by me. The biggest thing I did was the Environmental Audio System. I’m most proud of that. We knew that this game would need a wide dynamic range in order to convey a sense of fear and dread. Luckily, Bruce and Neil knew that, and were involved in pushing the game’s technology from an early stage.

We did a lot of things in The Last Of Us that we’d never done before. For example, we updated the Environmental Audio to support Reverb Portals, so now you can hear sounds that are being reflected down the hallway from an adjoining room. So if you’re playing the game with a Surround Sound setup, and a Clicker is in an adjacent room, you hear it before you get there. It’s seriously creepy, but cool. We also spent a lot of time working to dynamically alter the mix of the game, which meant that the overall mix was determined by the state of the game. So whether you’re in combat or exploratory mode, the mix changes, and we needed to create technology that would allow the game to drive itself in this way. So I worked a lot with Phil on making that work, and I’m very proud of how it turned out.

Phil: One of the things that made this project fun for us was that we were very involved in the emotional arc of  the characters. One of things I focused on a lot while directing my team was that things were to be told from Ellie’s point of view, and not Joel’s. He’s not necessarily scared of what’s happening around him, but Ellie is. So the soundscape mirrors that.

One of the things I’ve heard Naughty dog say in the past was that you were striving to achieve a sense of minimalism with The Last Of Us music. Can you explain how you went about achieving that?

Scott: To help us achieve the sense of minimalism, we chose to not start off with the big, emotional moments in the story, and worked backwards from that by adding in ambience, texture’s and tension music. We made sure there was a lot of space between everything, and only had a few sounds playing at one time. There are only 2 or 3 points in the game where we use an orchestra.

Another thing me and my manager did was play the game endlessly in the weeks before it shipped, and constantly took out some pieces of music. We were able to play the game from start to end, rather than in 20 minute increments, as we had to do whilst it was still in development. Having too much music in a game can make it feel like wallpaper, so we tried to get away with 20-30 minutes of having no music, which creates a more profound effect when it finally does come in.

One of the interesting things that comes to mind about critically acclaimed games like The Last If Us is how the team dealt with challenges. It doesn’t appear to be much wrong with the game, but you guys must have encountered problems. What were some of these and how were you able to solve them to produce what appears to be a near-perfect game?

Scott: My biggest challenge is always dealing with transitions between different music pieces. There are about 1200 segments of music in this game, and each of those require transitions, whether it’s overlapping music or varying levels of intensity during a fight scene. I’ve played a lot of games where the transitions are very jarring, so I spent time on this.

Phil: Bruce and Neil did a great job of directing this game, but at the same time, there was a lot of trial and error involved in making it. For example, we went through a lot of iterations with the Clickers. During the viral campaign for the game, we uploaded a video to Youtube which had Clicker sounds that me and Neil had made in my office, recorded into an iPhone. It was a silly clicking sound that I had made with my voice. But we evolved the sounds from that and it became much more. We also worked collaboratively with other departments, and it’s exciting when we see someone from the art department draw a Clicker for the first time, and it has no eyes. So we figured out that they use echo-location. So everything we do informs itself down the line, and we thrive on that challenge.

Derrick: We tried a lot of different things for the Clickers. Sound Deluxe DMG sent us some samples of ideas they had, which inspired me to try different things. We took some of their ideas along with what we were already working on and played around with it. When we finally decided to bring in some voice-actors, I thought of Misty Lee, whom I had worked with on Soul Sacrifice for the Playstation Vita. She had a great scream and was very emotive. She should get a lot of credit for the female clicker sound. All the weird dolphin-like sounds are her, mixed with with some sounds me and Phil made. The real challenge came when we had created all of this, and then played it for Neil, and he said “No, it’s too much“. We were like, ”We think it sounds scary. Why don’t you like it?“. But if Neil doesn’t like it, it’s back to drawing board, which is actually something I agree with. We end up doing better work this way because we constantly refined our process. We ended up sitting with Neil through-out one of the opening Clicker scenes, and he would give me pointers like “Mute this, solo that, give it more space here“, until finally he said “That’s it. Minimal“. The right amount of activity and negative space is what gave us the signature Clicker sound.

(Listen to Phillip talk about working on the Clicker sounds below)

Erick: The Stalkers and the Bloaters came on late as well, but luckily there had already been so much iteration with the Clickers that when I came on, the language had already been set. But for the Runner sounds, we had to cast more actors  because what we had at the time was coming off as a little campy. We needed more desperation in their voices, and not just a zombie-like rasp.

Phil Kovat: One of the things that Bruce stated early on was that the people in this game weren’t evil. It’s the Cordyceps fungus that’s driving them nuts. So we were always string to achieve this sonic dichotomy of “I don’t want to eat you, but I have to“.

Erick: That’s why the sound of the Runners is almost psychotic, like people losing their minds, and not just mindless noise.

Derrick: Yeah. During some of the voice-over sessions, we told the actors to imagine things like petting a dead cat and singing to it. It was the kind of visualization we thought would help for getting certain emotions out of them. I remember saying to one of the actors, Otep Shamaya, “Remember that scene in Texas Chainsaw Massacre where they find the dead grandparents in the attic?“. She says “No, but go ahead“, and I was like “Well, imagine you’re in the attic and you are (singing to them) as you rearrange their clothes“. And she goes “Ok, I got it“.

Jonathan: It would be great to say we have some secret recipe or closely guarded secret for how we make games, but that would be a lie. The reason our games are good is because we work really hard, and only hire like-minded people. No matter how harsh our critics are on the Internet, they’re never going to be more harsh than our co-workers. If you do something that isn’t 5-star awesome, you don’t even have to worry about what online critics will say. We’ll call you out on it. But even that doesn’t guarantee that we won’t make mistakes, because we do. If you played The Last Of Us a year before it launched, you wouldn’t recognize it. So it’s worth noting that we fail a lot at certain things, but our talented team makes us not afraid to break things, because we can fix them. So it’s better to go ahead and try stuff and then keep iterating until it finally works.

I’ve been here for 10 years, and I don’t think that we’ve ever missed a ship date for a game. But we actually slipped for the first time with The Last Of Us. From a corporate point of view, we don’t want to slip our dates, but we don’t want to ship a game unless it’s awesome. So when we realized that the game wouldn’t be polished in time, we had to push it. We had to choose between shipping a game that had some flaws and upholding the Naughty Dog tradition of  making awesome games and get Sony to push the date.

One of the problems we had with not only The Last Of Us, but the Uncharted games also is that these games are heavily story-centric. A lot of the exposition of that story happens in the dialogue. This creates nightmares for us, because game designers can create situations where they’re not thinking about dialogue, and yet dialogue is very tricky to create. If you and I are sitting across from each other and talking, the projection of my voice will be completely different than if I’m 8 feet away. This kind of depth can’t be created in a game by just adjusting volume levels. But then there are additional problems created by the level layout and the Environmental Audio System, where characters have to have conversations whilst they’re moving around. If they move into certain areas and get obstructed, then wouldn’t be able to  hear them talking. A lot of games don’t bother with addressing things like that. They just play all the audio in the center channel or don’t pan it, which is sort of a cop-out. If you do things like that, you might get around the problem of obstructions of sound in the environment, but it pulls players out of the reality of the space, and we didn’t want to do that. However, we got to a point where we almost took the cop-out route of pulling the dialogue out of the game’s environment. But at the last minute, I came up with a plan: We used a psycho-acoustic technique to trick the listener. We created a separate reverb unit that had a very short delay on it, and dedicated it to dialogue in cases where characters were obstructed. We called it the “Dialogue Sweetener”. This allowed us to create a tiny bit of reflection that was enough to trick your brain into thinking that the sound was bouncing around in the room. So if Ellie walks around a corner, you can still hear her as if she was close by.

A lot of games choose to aim for emulating how sound works in the real world, but that’s actually easier than what we chose to do for this game. Game audio consists of dialogue, music, and effects, but the human brain can only latch on to one or two at a time, as they compete for the same space and frequency spectrum. In the film and music world, those elements are fixed since they’re cut to a picture or a single audio recording. In a game, you can change the camera angle or just walk away to where you want. That’s way harder to work with. So The Last Of Us isn’t a perfect game. But thankfully, our bar is so high that most people don’t notice the flaws.

What kind of interdependency exists between the audio team and other departments? Did your work ever get delayed up stream because you had to wait for other departments to finish their work, like cinematics or coding?

Phil: Oh, yeah. Audio is inherently a post-production process. When games are in pre-production, there are a lot of moving parts. Any self-respecting audio professional would say “I’d like to see the media first and put my sound to a piece of work“. But in an iterative project like The Last Of Us we didn’t have the luxury to wait on other departments, who needed our audio for their play tests so that they could see what was working and what wasn’t. There were times when we would give them an audio-sketch of sorts to work with.

James: For dialogue, we grabbed some random people from around the studio and used the first draft script to record everyone, just to get the dialogue in. We essentially recorded the entire game with dummy actors so that the designers had audio to work with.

Phil: I’d like to point out something that not everyone might know about. In games, you have a lot of artists animators and designers. A level designer might only have one or two levels in total to work one, and an animator might only make one character or part of the game. But the audio team touches every single aspect of the game, from pre-production to the final mix. So there’s a level of work that might seem daunting to grasp unless you work in this field.

Derrick: Like Phil said, we try to front-load the game with dummy audio, but that’s hard when we haven’t seen any of the actual game yet. But we do try to gather audio assets by doing sound and field recordings. If I had seen the initial storyboards for The Last Of Us I would know that wed need have audio files for things like dirt, debris, blood, metal sounds, environmental sounds, insects, etc. We try to prepare such things beforehand, because at some point the other departments will come to us and say “We need audio to test what we’re doing“. We might say “But we don’t have any right now“, and they’ll say “Just put anything in there that’s close“. That forces us to refine the things that we do have.

How did you guys go about dividing up the work that stays in-house and the work that you contract outside people to handle?

Phil: The internal Naughty Dog team for The Last Of Us consisted of me and Bruce Swanson, Jonathan, James on dialogue and a scripter. That’s it. For sound designers, we mainly turned to Sony’s PD team. So Derrick and Erick come from there. I actually used to work there myself, at the Santa Monica studios before coming here. So we have a group of around 26 sound designers that we can call on to help out. But sometimes we will ask for outside help. We brought in Neil Uchitel on this project. He’s a commercial sound designer with great ideas. He records a lot of physical sounds like doors slamming and rooms shaking. There’s a scene in the beginning of the game where an explosion goes of in Texas, and the sound of the bedroom shaking is actually Neil jumping up and down and recording the sound of windows rattling.

Jonathan: When you’re making a game that’s as dynamic as The Last Of Us, the technology has to inform the sound design. If you’re told to “make a sound for something“, without knowing how it’s going to be implemented in the game, you can’t possibly do that. We like all of our sound designers to be on site and invested in the project.

Erick: If we do contract people, it’s usually for things like cinematic audio.

Philip: Right. We hired a company called Sound Deluxe DMG to do the cinematic work. They also provided the facility that we used for recording dialogue. They also provided foley services.

What made you decide to bring on Gustavo Santaolalla to score the game? Were you fans of his previous work, was he recommended to you, or did he approach you?

Phil: Early on in the project, Bruce Swanson was helping Neil with audio sculpting ideas, and music was a big part of it. They were looking at a lot of different composers, such as Carter Burwell who worked on “No Country For Old Men“. But Gustavo’s music was so inspirational, although we initially thought he wouldn’t be interested in working with us, because he hadn’t done games before. But one of our colleagues from the Sony PD Music Group said “just try calling his agent“. So we did, and he was interested. He sat down with Neil, got the pitch and said “when do I start?“.

Gustavo never really like to work to picture. He created musical themes and atmospheres on his own, and then Scott would go in and edit the pieces with Neil and pick what they wanted. He started sculpting various musical themes based on the imagery and ideas he was given by Neil, and it was amazing to see how he solidified our partnership with the very first piece of music that he delivered, which became the main theme of the game. That’s the kind of artist he is.

Scott: Gustavo delivered us a CD of 70 minutes of music before there as was even a lot of gameplay. Then Neil would go through it afterwards and say “This piece of music would work in this and that scene“. We also collaborated a lot with Gustavo on the instrumental soundtrack music. We spent a few days with an ensemble in Nashville, and paired him with a symphonic sound designer called Tim Davies, so that they could orchestrate some music.

How did the voice-over process work? Did you reach out to people about that?

James: We did a lot of the principal character auditioning on the Motion Capture stage. They sent out fairly standard audition requests. So the motion capture would be recorded and brought back the studio for review. Ashley Johnson, who plays Ellie, was picked first. Troy Baker, who plays Joel, came after that, after which the remaining cast was brought in to play off of them. Once the bulk of the principal dialogue between the main characters was done, Liam O-Brien helped to direct enemy interactions. At that point, we would have people suggest certain actors to us, and we’d bring them in for auditions. Liam also had his own network of actors that he knew, whom he’d pitch to us.

Did Naughty Dog license any music for this game?

Scott: Yeah, we licensed two Hank William tracks, “Alone and Forsaken” and “I’ll Never Get Out Of This World Alive”. For the “Left Behind” extension, we also licensed Etta James’ cover of Sonny & Cher “I Got You Babe“.

What kind audio resources are available to Naughty Dog? What were your favorites?

Phil: We have 6 audio rooms at our Santa Monica offices which were built in coleration with Chris Pelonis, as well as our theater. We mostly work in our own audio rooms, with a standard Pro Tools setup and plugins.

Scott: We all use Pro Tools. and share the sessions amongst our selves. We also have proprietary music scripting technology that Naughty Dog developed over the years.

Derrick: We used a lot microphones and recording gear from Sony’s PD Sound ground. They have a collection of gear that we can borrow from.

Jonathan: The main thing we need in our offices is a clean signal path, good speakers and a good room when recording. We use a really nice Dynaudio surround speakers setup, with a completely digital system that has no noise floor. We don’t need much more than a Pro Tools setup and our interfaces, since most of our audio recordings aren’t done at our offices. Our sound designers will do field recording on their own gear. PD Music has an incredible recording facility up north in Foster City, and their gear is phenomenal. So they record the music there.

Phil: When working on this game, one of the most important things was to have everything sound as natural as possible so that it would be more immersive. We weren’t too experimental on the sound design. We did do some pitching and editing of certain Infected sounds, but that’s it.

Derrick: One of the most valuable pieces of gear I have is Soundminer, which is a unique search engine program that comes with pitch shift functionality that was very useful to me. It enabled me to manipulate sound very quickly via the built in VST rack. Having my own equipment, I can travel around and record things whenever I want. So if i think “Gee, we might really some rock and debris sounds one day if we ever work on Uncharted again“, I can record it.

Jonathan: We don’t generally use canned sounds and sample packs for our audio. It’s all designed specifically for the game, which is something I think our sound designers like Derrick can brag about.

Derrick: If you’re a sound designer, your sound library is a part of your identify. It’s what you bring to the table in addition to your talents.

Erick: The main sound of the Bloater actually came from my kid. When he was an infant, I’d sit next to him and record him crying and making baby noises. So I slowed one of those sounds down, and that became a groan sound that we used. It’s interesting how you end up using sounds in unexpected ways. At best, I thought I might be able to use those sounds if I worked on a game that had babies in it. But it ended up being used completely differently.

Can you take me through the mix-down process for the audio on The Last Of Us?

Phil: When Naughty Dog made the switch from the PS2 to the PS3, they made a very intelligent decision to pretty much start over. When that happened, Bruce Swanson and Jonathan started a new audio engine from scratch. We’re still using Sony proprietary Scream tool for sound, but our unique metadata system is responsible for how it gets played. With this, we can tell the sound how to behave when it gets triggered in a game. This includes manipulating things like fall-off, reverb amount, filter cutoff and atmospherics. So the mix actually starts from when we put the first sound into the game. We try to think of how we want it to play and interact with the world. So our sound designers are tasked with having to intelligently place sound in this game.

Jonathan: The mix of the game isn’t driven from the output of the audio, because we’re not trying to replicate the way the human ear hears things. Rather, we’re trying to do something stylized, like Hollywood does. Anyone who knows anything about Hollywood knows that there’s nothing remotely realistic about the way that audio is mixed for a movie. But we’re working with a game, not a movie. So we not only need to set the mix knobs for the stems, but we have to set other knobs that teach the game how to mix itself depending on what happens. You can’t mix game audio by just spreading everything out on a mixing board and setting individual levels. By the time you realize that different foley sounds don’t match in volume between different parts of the game, you’d be screwed.

Was there anything you feel that you should have done better on this game?

Phil: I think we’re our own harshest critics, and we try our hardest. We’re getting some new technology for the Playstation 4 which will allows us to do new things.

Jonathan: Things will get better though. We’re still in the early phases of exploring what can be done with the PS4. But you can rest assured that the bar has been set with this game, and we’re not shipping anything inferior to that.

Derrick: From a sound design point, every game represents where I’m at with a certain style, and then I build on that for the next game. Ben Burtt, who did the sound design for the Star Wars movies, always says that he’s looking for the perfect explosion. I’m looking for the perfect punch, and I think I got pretty close on this game. Not to say that the “perfect punch” is a sound that can be used for every punch, but I think I got pretty close using this particular punch sound to tell a story in this particular game. But I won’t be pulling The Last Of Us punches for the next game. That’ll be a whole new creation process.

James: You can break down dialogue into spoken words and effort sounds like climbing and punching. Within efforts, we have to build different system to make them sound natural. So if you’re pushing an object, you’ll hear the starting sound, followed by some loops like grunting and huffing, followed by final sound when you stop pushing. As a part of fleshing out that system, we create probability systems that ensure you don’t grunt on every step. So we intend to keep working on systems like that to make it sound more natural. As we develop more games like this, we have to create complex dialogue systems to try and model realty. So each time we do that, it has inherent problems, and we have to work hard to make it sound more believable,.

Jonathan: Touching on that, it’s worth mentioning that the conversation system in our dialogue manager can be a double-edged sword, because the more you allow characters to interact verbally, the more telling it is when they break form or do something unrealistic. We call it the “uncanny valley”, which means that we’ve made things look so realistic that anything which doesn’t look real will stand out. When your dialogue manager is so sophisticated that you can feel character’s moods and feeling, the in-game Artificial Intelligence can’t afford to make mistakes. If the AI makes a dumb move, it’ll be telegraphed through the dialogue. So one of our biggest problems with raising the bar for dialogue is to create a smarter AI, because it can easily do things that are wrong. Not that the AI is malfunctioning, but we depend on it to think like a real creature. So if the AI makes a bad decision, then sound is the first indication of that. For example, a character might fire a weapon randomly or someone might call out an alert to a character that he shouldn’t know the name of plot-wise.

The dialogue system in The Last If Us is amazing, and James should get a lot of credit for working with Jason Gregory on it. We never had anything remotely as good for the Uncharted series. The next Uncharted will benefit heavily from this.

James: We shipped this game with close to 70,000 of dialogue, which is huge amount for a non-RPG game this size. We also built a context awareness system so that enemies can call out your location when you’re fighting. In order for that to work, you have to record every single object and location that are in the different environments. “He’s by the van!” or “She’s by the bar!” or “She’s across the street!“. So once the levels were fleshed out, we’d have to go through each area and take note of the fact that we might have a store, an office, a street and an alleyway present, with vans and trucks parked outside. We’d also have to take into account one car parked over here and multiple cars over there. Also, when the enemy spotted you, they have to call out that you were standing next to a car, and if they can’t see you, they say you’re standing behind it. Not only did all of that dialogue need to be recorded, but it had to be recorded multiple times with variations, so that it didn’t become repetitive and unnatural. Our poor voice actors were screaming “He’s over there” and “She’s by the car“, for 4 hours over the course of 5 recording sessions. I’m sure they all hate me, haha. But that’s what it takes to model this kind of thing, and as games continue to model reality even more, we’ll get into this kind of territory even more. Sorry voice-actors….

What made The Last Of Us different from past Naughty Dog games, like Crash Bandicoot and Uncharted, and how will its success change the future of games you make?

Phil: Instead of us being reactive, we participated more than you might have expected. Prior to this, I did cinematic work for Jak & Daxter, and then I worked on Uncharted 1 and  2 . So I’ve seen how Naughty Dog has progressed. We started off with Crash Bandicoot, which was very cartoony and platformy. Then we took it to another level with  Uncharted , which was aimed at being realistic, though still a bit pulp and platformy. But taking it to serious level and making people cry is what we did with The Last Of Us. So this at kind of emotive work that we did has changed how we approach making games.

Derrick: I think this game sets a precedence for improving how we work together at Naughty Dog. We don’t have to deal with other team members saying “We’re done with our work. Now you guys go make some sounds for this“. We get to almost design things alongside other departments, which we appreciate.

Phil: When the director comes up to us and says, “we’re depending on you guys to make this work“, and isn’t just talking about fixing a problem, but actually create some emotional context, like with the Clickers, it’s special for us.

Erick: After having worked at a bunch of different studios, I can say that Naughty Dog is the most collaborative one that I’ve worked at.

Phil: There’s a part in The Last Of Us: Left Behind where Ellie and Riley come across a broken fighting arcade game, and has to Ellie play the video game sequence in her mind. So the health bars and character names appear on screen , but when you think about it, that entire scene is based on sound. Erick worked very closely with one of our designers to make it work.

Erick: Yeah, that was an iterate process that was very audio-driven. The whole thing is her imagine how the game happens, so we had to use all kinds of Mortal Combat-like sounds to sell the moment. It worked out well.

Derrick: There were moments like that on Uncharted 2 as well. In fact, one of those lines was thrown out of the game initially. During the museum heist, when Drake’s climbing up the wall, Flynn talks into his receiver and says “There’s a man above you! There’s a man above you!“. So Drake grabs the guy and throws him off the building and Harry Flynn says “There’s a man below you! There’s a man below you!“. I love that line. But then one day it was taken out of the game by someone. So when one of the animators was walking by, I said “I can’t believe they took out the Flynn’s line!“. He replied, “You want that line to be there?“, and I was like, “Yeah, it was a funny line!“. So he goes, “I’ll be right back“. He leaves and comes back a minute later and says with a grin, “It’s back in“. So everybody values each other’s opinion here, and I think that sort of thing will continue to be a part of how we work.

Jonathan: I’ve been here since Jak 3, and it’s been really interesting to watch the evolution of audio that’s taken place at Naughty Dog. I might have a job title today that’s called “audio programmer”, but I was never hired as that. My job interview was grueling, since they’re very picky about who they hire here. But we got around to talking about audio somehow, and I remember mentioning that I was very passionate about it. At the time, no-one around here wanted to work with audio and I was like “I love doing that!“. During that time, there wasn’t a lot of audio technology at our disposal, and audio memory was a precious resource. Also, the games that we made back then were more cartoonish and fantasy-based, with beep-boop-boink sounds. Things took a turn with the Jak & Daxter games, but Jak didn’t even talk in the first installment because we didn’t know how to stream audio at that time. So one of the first things I did when I started working here was to re-write the streaming system. I knew we weren’t streaming anywhere near what we could, and I sped up the streaming format so that we could move from using MIDI music banks to actually streaming audio in the games. In fact, one of my first meetings was a talk about how we were running out of memory, and Bruce Swanson was like “We need to save memory for the MIDI banks“, and I said “But we can free up all the memory banks and just stream the music“. He replied “No we can’t do that. The streaming system isn’t fast enough“. I’m like “I can make it fast enough“. Everyone looked at me and said “Are you sure? The guy who worked on it previously said it was impossible“. “I can do it” was my answer. So one of my first tests was to re-write the system by a certain deadline, and when we shipped Jak 3, we were able to do so with real streaming music.

It’s been  a slow progression, but I think Naughty Dog eventually realized that the game’s are cooler when the audio sounds better. So when they reset the whole company. discarded our old code and switched to C++, the company came to me and said “You’re our only audio guy and Bruce really needs you right now, so we’re gonna have you work with him as long as he needs, and then we’ll pull you back onto the game team when he’s done“. Seven years later, I’m still on the audio team.

Like we said before, sound has a history of being treated as something that comes in at the end of a game, when the programmers hand over over their work to the audio team and say “Here, add your sounds to it“. Bruce doesn’t work that way. He said “If you want a sound, you come and talk to me and I’ll give you a sound that’s designed for your particular thing“. So as audio became more important, we grew more collaborative with the other disciplines, and now they don’t think of us as guys at the end of the pipeline. Sadly, there are still companies out there who don’t work this way. Erick has probably dealt a lot with that, haha.

Erick: Yeah, it’s not fun….

Haha, I hear you. Hey guys, it’s been quite an extensive talk about your work at Naughty Dog and I’m super happy you took the time to speak with me. I’ve learned tons about how your audio team works and I’m sure many others will to. I hope everyone wants to check out your website, Twitter and Facebook pages to stay abreast with whatever Naughty Dog does.