WETA's Joe Letteri Helps to Bring Middle-earth to Life One Last Time

WETA Digital didn’t become the leader in large-scale cinematic visual effects without a strong hand at the wheel: Joe Letteri, Oscar-winning senior visual effects supervisor.

Letteri has the led the effects charge into Middle-earth for Peter Jackson's J.R.R. Tolkein adaptations, culminating with The Hobbit: The Battle of the Five Armies, as well as King Kong and The Lovely Bones. He was also a guiding hand in the visionary work appearing in James Cameron's Avatar, Zack Snyder's Man of Steel and the relaunched Planet of the Apes franchise.

Speaking with SPINOFF ONLINE, Letteri recounted his experiences working on the final film in the Hobbit trilogy, and offered a tantalizing glimpse at his future work.

Spinoff Online: For The Battle of the Five Armies, what was the hurdle? The thing where you said, "If we could just accomplish that to everyone's satisfaction, half the job's going to be done right there."?

Joe Letteri: That would be the battle. [Laughs] Since it was the title of the movie, as we found out! And yeah, since it took up the whole third act, that was a major piece of filmmaking for us. It was very nebulous at first, to understand what the shape of the battle was. And because so much of it was virtual, we really had to sketch it in first. So we laid the battle out on the virtual stage. We brought Peter to the stage with a virtual camera, got him to walk around and kind of sketch out where he thought the different pieces of action would happen. And also how we wanted to frame shots.

You could look up towards the hill where Dain's army arrives and we could see, "OK, if we're going to shoot this on a 75 and we're down in front of Erebor, are they close enough? When you put the Dwarves down on the hill, can you see who they are? No? Let's move the hill closer." We were reshaping the whole battlefield in this really rough form, just to get everything happening. When you pan over now and you're lined up and you see Thranduil and Bard at the gate, how does Dale look? Does the composition look good? Nope – move the city over. Can I see Ravenhill when I need to be talking about Azog using it as a command center? Nope – move Ravenhill around.

And so we were constantly reshaping it as these requirements came up, and as we started getting new information about new pieces of the battle, and then putting that together into one consolidated whole. So that was a lot of pushing and pulling, just to get the battlefield itself built, because the battlefield was completely digital. And then we had to get the armies going. You had a lot of very highly detailed, chaotic action, so the choreography for that was pretty intricate. A lot of stunt work, a lot of performance capture. But you know, in performance capture you are limited. You can get about maybe six people in close contact fighting like that at any one time. And you have a pretty limited run.

You know the stage is maybe about as big as a basketball court, so there's not a lot of room for doing big, broad action. So we have to do it in pieces, so when Peter wants to shoot down the line and see hundreds of people clashing, you have to do that piece by piece by piece and then figure out a way to stitch it all together; so the choreography and the animation that goes on top of it was pretty intricate and pretty complex. And they have to figure out how to light it all. And from a CG perspective, there's the rendering side of it, which means when you're going to have these big, massive scenes with thousands of characters, with armor and costumes and hair and dirt and grass, the complexity of the scenes is enormous and we have to just throw a lot of artistic talent, to be able to light it in an interesting and cinematic way. And just a lot of computer power, to be able to push the shots through.

You've worked as closely with Peter Jackson as anybody has. What's unique about working with him? What's the thing with him that you don't get with any other filmmaker?

Well, Peter's very spontaneous. He'll set things up in the moment and he will just react to what's there. He's very flexible about what he's shooting and rearranging things and trying to make it work. That even extends past the point of where he's done something and is satisfied with it. If we look at it while we're doing it and think that we can come up with something that might work a little bit better, he's willing to look at that and say, "Yeah, if you've discovered something here that works, let's go for it."

Is there a subtle little thing that you and your team worked on, that most audience members wouldn't notice until maybe the third or fourth time they watch the movie, that is an effect that you're particularly pleased about? One of those almost invisible effects that you pulled off?

Oh, boy – there's probably a fair amount of those, but I have to say, one of the ones I'm still curious about is our digital characters, especially Azog and Bolg, because I still get people who are asking me how we did the makeup for those guys, not understanding that they are fully digital characters, I think. To me it's interesting that we've reached that sort of threshold, especially after having worked with these characters for a couple of years now, they just kind of show up in the scene for us like actors. They might as well have been on a shoot. They just show up and everyone knows how to make them work and they drop right in.

Was there a technological or stylistic breakthrough on this particular picture? Something you hadn't done before, or a new way that you approached something that's revolutionary?

Yeah, for dealing with the lighting and the rendering we wrote a new renderer, that we called Manuka: if people aren't familiar with rendering, it's how you compute all the lighting that happens in a scene from sunlight bouncing off of clouds and hitting the ground and bouncing off the ground into a characters eyes and into the camera. Hundreds of millions of calculations of all of these different rays of light, and the scenes have gotten so complex over the years that we decided a few years ago that we needed to embark on our own project and just write our own rendering so we could basically work with these new complex algorithms, because we knew we had these big scenes coming.

And we did break ground with it on Dawn of the Planet of the Apes this year, so it got its first use in those shots, but we used it full-bore on Hobbit. Probably about 90 percent of the film was rendered through that, so the big battle scenes. The close-ups of the Orcs and Azog and Bolg and all of our digital doubles – these are all things that we can now just do, even more realistically than we've been able to in the past, because of this new lighting software.

How do you divide your creative attention when you've got the Hobbit films in progress, and you've got another film like Dawn happening at the same time, you've got Man of Steel wrapping up? Are you at a point where it's easy to divvy up where you focus attention on a given day? How's it working out?

Yeah, we have it pretty structured, because the team that we have at WETA that is doing all these films, we've been together since the Rings days, so everyone that's supervising the films or aspects of the films, have all come from working on different sequences. When we did Rings, these were the same people who were doing sequences, or King Kong or Avatar, they had their own sequences, their own chunks of the movie. And we've just followed that idea where now, yes, they're separate movies, but we still work in the same way.

I'll sit in with all of the visual effects supervisors and look at what they're doing on each of the films and they're always talking to each other, so that we know if anyone's got anything innovative that's working in one place, we want to make sure it gets used throughout the facility. But that's probably the real strength is the fact that all of us go back 12 to 13 years together as a team, and it's really allowed us to keep that cohesion, even though there are a number of films spread across the work right now.

Having been a part of that pioneering wave of digital effects, today we watch the credits of a movie that has a lot of effects sequences and you see 12 different effects houses have been called in to do different things, it's just a ubiquitous part of the industry now. Give me your take on the state of the industry as it stands today, and then what you think you and your team specializes in, what you at WETA think you do better than anybody else.

A lot of films do get split up these days, and we do work on parts of films as well. There's quite a broad range of work going through WETA, but what I like to do and what I prefer to do is to have the whole film at WETA. This year we were able to do that with Dawn of the Planet of the Apes and with Hobbit, because it allows us a measure of creative collaboration with the director that you can handle any problem that's thrown in any direction on the film. I don't ever like to be in a position where I'm working with a director, and he or she is asking for something and I can't say, "Well, I can't help you there, because we're not doing that part of it." I like to look at the film as a whole and say, "OK, if this is what you need for the film, then we can make it happen."

And so that's to me the best way to work, if we can just keep the film together and do it that way. And that's been especially true with a lot of the films we've worked on like Dawn of the Planet of the Apes that are very character-driven. It would've been really difficult to split up a show like that where so much of it depends on the performance of the characters. It would be like saying, "We're going to take the same character and for half the movie we're going to have one actor portray him, and for the other half we're going to have a different actor portray him." Unless there's a creative reason to do that, like it's a different period in history, that makes no sense. Why would you do that?

And so that's the approach that works best for us. And of course, working with Peter on these films – because we have been creating the Tolkien world for the last 15 years – it has really made sense to keep that all together. That way we can develop the look of the world, and we collaborate not only with Peter but with [conceptual designers] Alan Lee and John Howe, and a lot of the art direction, and Richard Taylor on the character and designs We have a team that has been working on these films since The Lord of the Rings and so everyone is very imbued with the history and the culture of Middle-earth, and it allows us to draw on a lot of resources to bring things to the screen in a way that's integral to the story.

This has been, as you've mentioned, a long journey throughout this particular franchise. And I see, when I saw the movie and watched people reacting to the movie, it's emotional at the end, particularly to know that this is probably, for all intents and purposes, the last one. As you think about the journey for you, personally, what comes up?

It's great to look back on it. I'm just really pleased with the way all of it has turned out. To me, what's most memorable is that we've gotten to create this world, and we've gotten to create some fantastic characters within the world that have become part of the film in a very integral way. When I think back on Gollum and how that was to see him come alive for the first time, but also to know that this isn't something that just kind of looks good or looks interesting – he's moving the story along. He's pushing the other characters. He's making things happen. It's like, "Wow, how great to have characters that can do this, that are really just based purely on performance." So you get to work with a great actor like Andy Serkis to move that along.

Or you can look at the Tolkien stories and read the books and you see Alan Lee's artwork and John Howe's artwork and to be able to work with Alan and John and say, "How do we make this come alive? How do we step into this and bring audiences into it?" I've been really fortunate to be able to do that, and it's great to be able to look back on it and say, "Yeah, there's a world there and it kind of holds together." And I especially like what Peter has done with the Hobbit films, because I know there are a lot of people questioning, "Why three films?" But he took great pains to tie in The Hobbit into setting the groundwork for what happens on The Lord of the Rings, and so you have this really interesting mix of the world now and this feeling of it being complete. So it feels like a good place to end it.

You mentioned Andy Serkis: Here's a guy who is really leading the charge to the wild, wild west of digital acting, and you've worked with him in different capacities now. Tell me what it's been like to see that sort of occupation come to life to the degree that now pretty much every actor's probably going to have to do some form of that at some point in their career going forward, and to watch Andy pioneer that craft?

It's been great, because what Andy really showed is that you can have a performance that is so powerful that it doesn't matter if you're seeing your own face on the screen. And that's been really hard for a lot of people to understand. But Andy's fearless that way. We can make him look like Gollum. We can make him look like a chimp. It doesn't matter. You'll still get a performance that moves you. And that's, I think when actors talk about performance capture and they get worried about it, when they start to understand that aspect of it, they realize that there's something there that's worth exploring, because it strips the art down to its essence.

Did you get to work with him when he was doing his second-unit directing duties?

Yes. It was great, because Andy being an actor was really great in just kind of setting up the scene and giving the actors other things to do that were interesting. It was a very collaborative process. It was great to work with him on that.

What would you call the gateway drug to your occupation? What was the thing that fascinated you as a kid or as a young man that made you want to kind of tear apart and dive right into it?

You know for me it was a mix, but if you really go back to the films themselves, it's probably the original King Kong, the original Harryhausen films, some of the old science fiction films where you just wonder, "How did they do that? What did they hang up there to make it look like there was a flying saucer?" Just really simple things, but then learning about green screen and illusion.

It's an interesting mix, because I came at it with that interest, more in sort of the magical aspects of it, the perceptual aspects of it. I was more curious about it, not really thinking I was ever going to be going into film. I was much more interested in science, but I was also interested in photography and all these things started to come together and then at one point for me, it was computers. I started getting interested in computers and that lead to graphics and the idea that you could make pictures with computers, and it suddenly all started coming together. It's like, "Wow, you can do this. You can make anything you want." And that's kind of like, "OK how do you pull this all together?" Well, you need to know a little bit about physics, you need to know how things move. You need to know how light works. You need to know mechanics. You need to know how to program.

And it was one of those things where, it's not about drawing, it's not about photography, it's not about sculpting. It's not about any of those things. Yet, it's about all of those things. And it's really kind of an interesting art form that's really kind of only possible because you can use a computer as your canvas.

Were there one or two particular giants or legends in your field that you got to have a really meaningful encounter with or exchange with?

I was really lucky to start off at ILM working on Jurassic Park with Dennis Muren, so that's probably the best start you could ask for.

I know you've got some more big projects on your plate coming up. As far as details, you probably got to keep some things close to the vest, but what are you able to say at this point about Batman v Superman: Dawn of Justice?

Yeah, we're doing one or two sequences on that. Yeah, I can't really go too much into it, but we've enjoyed the collaboration with Zack [Snyder] ever since the first film and so yeah, it's great to be able to get to carry that on.

Can you characterize philosophically what you're carrying forward from what you did with Man of Steel to this one?

No, not really. It's new, so probably can't get into it too much.

And then there are the future Avatar movies. Everybody's dying to see what happens with those.

Yes, yeah, exactly. We know the scope of what's coming, and so we're constantly moving in pre-production on those. Like, a lot of our stage and lighting tools and our real-time tools are constantly in progress, and we're working with them all the time. To the point where a lot of them were developed and we convinced Peter to let us test them out on him when he was doing his virtual production work for Hobbit. He agreed to be the guinea pig for the next generation of software on that. It went mostly pretty well. [Laughs] A couple of rough spots, but yeah.

I wonder, knowing how ambitious a filmmaker James Cameron is, are there things you know you're almost going to need to invent, or are you going to have to break some technological barriers to get where he wants with the next round of films?

We hope so. But we've already been discussing with Jim what he's looking for in those areas. And we're underway on that R&D already, because it's a multi-year effort to get where we're going to need to be.

How does 3D in general affect your visual effects work? Are they an easy mesh? Do you have to do certain adjustments to make things work in 3D? How does that all kind of shake out?

For us, I prefer if we're doing 3D to be able to do it as native 3D. So we were fortunate this year and we did Dawn of the Planet of the Apes and Hobbit in native 3D. I think it gives you a more realistic result, because everything is physically correct. Conversion can be done very well these days, but when everything is just completely correct, I think it's more believable and more comfortable to your brain. So I prefer to do it that way.

It does make our life as visual effects artists a little more difficult because in a purely 2D sense, you can get away with a lot of things if you're not so worried about the spatial aspect of it. Probably a good example is on Dawn of the Planet of the Apes, where we had the actors riding horses. So it was the ape actors in their performance-capture suits, but they had to have a saddle and they had to have stirrups, but the apes were riding bareback with no stirrups. So there's quite a lot of paint-out that had to be done. First, the actors are bigger than apes. Their legs are longer, but then we got to paint out the saddle, got to paint out the stirrups, so painting that back on horse with all the fur and muscles movement is a ridiculously difficult proposition. And I've got to give our paint department a lot of credit for that.

And then if you realize you kind of have to do that with stereo, with everything – if you get it looking right in this eye, but it's not right in this eye, and you get it looking right in this eye, but it's not right in this eye, and you're constantly bouncing back and forth between the left and the right eye – it's an extremely difficult proposition. Whereas in 2D, you can get away with a lot more, you can cut and paste bits of horse on there. And so that's where the work becomes more complicated. And ironically, what it means is it's the work that requires all the hand work that gets more difficult.

But on for example, the animation side, you only animate the character once. You just photograph him from two different cameras. Same thing with the lighting. You light it as if you're just lighting the world, and you photograph it from two different cameras. So those things that are technically more challenging are, ironically, easier to deal with because you just let the computer take care of it.

When you watch someone else's movie, with a lot of visual effects, are you constantly deconstructing it, or do you immediately get sucked into the world that the effects are creating?

It's a little bit of both. I go to the movie to watch the movie, but I also understand what's going on. It's hard not to – I've been doing this for too long to ignore what's in front of me. But generally I understand that everyone is doing these things in service of the film, and so that's the whole point. I try not to be critical so much as I'm just aware of what's going on.

When you get up in the morning every day, what keeps you excited to go into the office and get to work?

It's always the newness. Every day we create something new, which is kind of cool. That's the interesting thing about this business – and one of the things that makes it a little difficult – is even if you're doing the same thing, even if you're doing 25 shots of Gollum, yes, it's Gollum, but every shot is new and unique. So every one you still have to look at it and get it just right. And when you do get it right, you can sense that it clicks. And then you move on, because you have another shot to go on to.

For the super sharp-eyed audience members, is there one little thing that you'd like to point to in this film where you'd say, "Look for this, because this may be a small thing, but it's really cool!"?

Oh, boy, that's a good question. I can't think of anything. There are so many moments like that, but I don't know. I guess one that's coming to mind right now is when Azog slips under the ice. I think that's a really nice-looking shot, because we really tried to play with the refractive effect of the water to kind of – we actually cheated a little bit to give you the sense of the story, so you could actually get a clear read on him, but we also played with the murkiness of the water to give you that sense of him being swallowed up. So that one actually went through a few iterations, just to get the right feel of watching him go away, but still being able to stay with him the whole time.

The Hobbit: The Battle of the Five Armies is in theaters now.

Masters of the Universe: Kevin Smith Hypes Voice Cast Announcement

More in Movies