It taken countless hours to fill three consecutive films full of the most realistic ape characters ever seen on screen for the revived Planet of the Apes trilogy, but for Weta Digital director Joe Letteri, one of the most accomplished visual effects artists in the industry, it was all worth the effort.

With the trilogy’s final chapter War For the Planet of the Apes now available on home formats, Letteri took a moment to look back on what will inevitably be looked on as one of the most significant achievements in the effects field, and one that will spark pioneering breakthoughs yet to come. And that’s saying something, considering that Letteri and his team at Weta have already changed and re-changed the FX game multiple times over, most notably with their work on Peter Jackson’s Lord of the Rings and Hobbit films and James Cameron’s Avatar.

As he takes a breath before digging in deep to his upcoming projects Alita: Battle Angel, for producer Cameron and director Robert Rodriguez, and a planned series of new Avatar films, Letteri joined CBR for a look at the totality of his Apes experience, the particular magic of Andy Serkis, and what further bar-raising visuals the future might hold.

CBR: When it comes to the Apes trilogy and the scope of it, I'm sure when you started you expected that if the first film was successful, there would be more than one movie -- so what was the vision going in for how your team at Weta was going to approach your end of it?

Joe Letteri: I think the most important part for us was to take the idea that it was an origin story starting in the modern day, and figuring out how to make those chimps look realistic. They have to be indistinguishable from what you could go out and photograph. But then to start bridging that gap towards them getting intelligent -- and especially the ability to speak, because Caesar starts to speak at the end of the first film.

So even then we had to make a few design choices, like giving him a slightly smaller muzzle so he could articulate better when he did have to do that, and then just from an expression aspect, bringing in some of Andy's features, especially around the eyes. Because you're watching Andy's performance and you want to recreate it, and so much is in the eyes.

So what we did is surrounded him with chimps that were completely based on real chimps that we could photograph and study, and then fit Caesar into that with his slightly more unique design and treat that as part of his personality, like he was born with the drug in him, where he started to have a little bit of this intelligence. And that gave us the springboard to give him those unique properties.

And then as the films developed, like in two and three when the other apes had to start showing intelligence, we just made subtle changes between each film. And we hid it in the aging process, right? So we made their muscles slightly smaller so that they too could speak, and made their eyes a little bit more expressive. They still look like the same character, because you're 15 years later and you expect some changes. But it gave us that grounding to be able to show the enhanced intelligence.

Tell me about working with Andy, who cracked the psychology of Caesar for each film, and incorporating his performance into what you guys were trying to do visually. That merger is so seamless in these films.

We spent a long time on the first film working with Andy and working with Terry Notary and working with all of the other Apes actors like Karin Konoval who played Maurice, to really understand ape behavior. Because a lot of it is very physical. Apes have longer arms and shorter legs than we do, so the actors have to learn to move with arm extensions because apes are quadrupedal. And that physical performance has to become the basis of it. Because if you make the apes look completely realistic that they stand up and just walk around normally and you capture that, it looks like a man in a suit. It just looks like what it is, because the performance comes through so faithfully.

So that physicality had to be like the groundwork that you didn't even think about. That's what the basis of it was, and everything else got layered on top of that. So we spent a lot of time working with it interactively to understand that, and to really talk about the psychology of the character and where the arc was going. But by the time we got to the third film, they all knew what they were playing.

So for us, it just became a question of just paying attention to the details -- which is good, because in the first film we were wondering, "Do they look chimp-like enough? Are they moving the right way?" We were focusing on the translation aspect of it. But really, by the time we did the third film, we could really just delve deeper into the artistic side of it, really making sure we got the nuance of the performance. Because that was really important for War.

Was there a new breakthrough each film? Was there a level that you crossed every time you made one of these movies?

Well, the biggest breakthrough was the first one. Just being able to take performance capture out onto the stage, right? Because before that, like when we did, say, Lord of the Rings with Andy, he'd be in there with the actors, but we'd have to take him back to a motion capture stage with all the cameras around and say, "OK, Andy, here's what you did. Can you do it again?" And capture it and put it into the plates. Because performance capture is really a bunch of cameras shining a bunch of lights at the actors to try to see their performance from dozens of different angles so you can reconstruct it. That doesn't play well with the photography, because you're throwing extra light into the scene, and vice versa, the lights that are lighting the scene can confuse the performance capture cameras.

So our big breakthrough was figuring out how to do that in Rise. And once we had that part cracked from the capture aspect of it, it just became more and more about making this transportable anywhere that you wanted. Because [director] Matt [Reeves] had us going out into the woods where the conditions were just really remote and harsh, so we had to just figure out, technically, how do you keep the weather from affecting this? So it's pretty delicate gear, you're trying to capture movement down to a sub-millimeter level -- and it's raining! Like in War, it's snowing and the actors are rolling around, and it's just really physically demanding. So we were able to evolve that side of it through the three films.

So when we call it performance capture everyone thinks of it as, "Well, here's Caesar's performance," but really you're capturing the actor's performance. The other side of it is the translating that and making the character on the screen, and there are a lot of aspects to that. We've gotten better doing muscle simulations, we've gotten better at doing the fur dynamics, the way light scatters in the fur, the way debris accumulates in the fur, rain, snow and all those sorts of things -- better at creating the physicality of the lights in the world. It's called a renderer, which is the software that actually traces what all of the light rays are doing in the scene, and it much more accurately mimics what's happening in the real world.

And then we broke that out completely for War, so it was done 100 percent with this new renderer that’s called Manuka. So there are all these breakthroughs that add to the realism, but they all have to work together, right? If any one part of it falls down, you tend not to believe any of it.

As more and more actors become conversant in this style of acting that's necessary for performance capture, why is Andy still the top of the heap? What is the secret that you see in him approach?

I think, in a word, Andy is fearless. He is totally committed to the character. And to him, the lack of a costume doesn't really matter. In his head, he is Caesar. And he also has a lot of confidence that he knows that we're gonna give him that back at the end of the film. If he sees it, it will be his performance, but it will be Caesar doing it. And I think a lot of actors now that they've seen the process are becoming more comfortable with that.

In the early days Andy had a lot of faith to just let us do that right off the bat with Gollum. But when you work with new actors, they have a little bit of hesitation. "It's not my face, what's gonna come through, is it gonna look like me? Are you basically gonna screw this up?" And now that they've seen enough of the process, that hesitation goes away. So a lot of actors just walk onto the stage and they're comfortable with it now.

And my feeling from talking to a lot of actors is, to them it's like a workshop. They're just in the heads of the characters, the characters are in their heads, and when they're working with each other, they're just working with the actors, and they're not looking at the rest of it. It's all very in their minds. And a lot them actually like going back to those old workshop days and just stripping it down. So we've found a lot of acceptance from the actors that have picked it up and have been working with it.

Page 2: [valnet-url-page page=2 paginated=0 text='Working with Batman director Matt Reeves, Plans for Alita and Avatar']

Matt Reeves has really made an amazing reputation for himself on the Apes films. You've worked with a lot of different directors: what's special about your working relationship with Matt? What struck you about him as a filmmaker?

Look, I guess any good director does this thing where they are totally immersed in the character and totally immersed in the story. And I think that is Matt's strength when it comes to these films. He works with the actors, and he gives them the space to create the character, and he does not stop until both he and they have really pulled out that last bit of nuance that he wants for each scene, and that they want. It's a really collaborative process, and Matt just makes sure that everyone has the space to make that happen. He really just stays focused.

Do you have any expectation that you guys will work together on his Batman film?

I have no idea. I'm not even sure if Matt has started writing it yet. Would love to, but again, I have no idea, really.

Like the actors, you at Weta immerse yourselves in whatever world you're trying to make come alive. Tell me about spending all this time studying apes, and what were, for you, the big fun revelations of spending all that time looking at these creatures?

Well, it's interesting, because apes are very much like humans. They're very social creatures, but the also, like humans, have a very violent, dangerous side, which you may not think about because you're used to seeing them in captivity or in these documentaries. And it gives them a surprising explosiveness that plays well in the drama.

There's a scene in Dawn where Koba is in the armory and he's trying to scope out what's there, and it starts to dawn on him what he's seeing. And two guys walk in with the guns, and he's looking mean and menacing, and he just switches into circus clown mode and totally disarms them. And it's just a brilliant performance, because he's playing to their expectations. He knows what they expect to see, but yet you see in his eyes: he's gonna do damage.

And then you come to the next scene where he just plays on that and walks in and does the whole thing with the whisky and just basically gets himself right in there. And they're laughing, thinking he's a clown, and then he just switches on a dime. And that is inherent in chimp's personalities. You can't get close to them. They are fast and they will grab. And you don't expect that, but when you really understand that, it opens up all of these dramatic possibilities. And Matt really took advantage of that in these films.

Is there a special sequence among the three films that you can point your finger at and you say, "My team did some spectacular work in this particular scene that we're very proud of?"

I'll tell you, the first shot that we did that we showed to the studio of Caesar was a shot that they wanted for the trailer, where it's after Caesar decides he's going to stay in his cell with the chimps and not leave. And he's sitting there, and he's just watching the two guys who are guarding him go by. It's just in his eyes. And you start to see suddenly that he is now thinking about himself in a different way.

Up until then he thought he was a human. Now he realizes that he's an ape in a human world, and he's got to figure out how to make this work in a very dangerous situation. And it's just in his eyes. And it's the first shot we rendered of Caesar, and it's the first shot I think when the studio saw it, they said, "Okay, this is gonna work." Because up until then no one was sure.

But it's those character moments that we always look for. The big, dramatic moments, in some ways are easier than those small moments where it's just happening in their eyes, where that little light flicks on. And if you can't convey that, it's dead. You don't get to take it to the next level.

Working on a series of films like the Rings and Apes trilogies and have that epic experience, how does that affect you as a filmmaker? When you get to the end of it, is there a sigh of relief? Or is there a sense of, "Oh, gosh I'm gonna miss this?"

Yeah, there's a bit of that. Like when we started Rise, we knew it was the beginning of a whole series. And we knew that it had to in some way tie back to the original film. Even though it wasn't really gonna get there, there's still that feeling of, "We know apes are gonna be dominant, they're gonna be on horseback, humans are gonna become mute, and there's a Statue of Liberty somewhere.”

So we had no idea how we were gonna get there, but we knew if the first film worked, which I totally believe from the first time I read the script that it would work, that you'd be able to carry it on. But a trilogy seems like a natural stopping point, and I think Matt played that really well, especially with Caesar's arc. So yeah, there is a feeling like, "OK, we've told that part of the story. Now it's time for a breather." I hope there are more, because I like this world that we're creating. But yeah, it's just kind of a natural breather.

Tell me a little bit about what you can say about your approach to Alita: Battle Angel. That's another one where it's property that people know, -ish.

Yeah. I can't really talk too much about it other than to say that it is a character-driven piece, and we're using just a lot of the tools that we've gained from doing all these other films to really make that be the heart and soul of the film. Which is what I love doing. I love doing these character-driven effects pieces.

Is there, on a technical level, some special achievement that you're going to try to get to?

There might be, but yeah, I'll have to wait ‘til it comes out to talk about that!

And then you’re gearing up for a return to Avatar. Has Jim [Cameron] laid out the totality of his plans for all the Avatar movies he's got up his sleeve?

We have a pretty good idea of where we're going with it. And the thing to me about Avatar that I really love was the way Jim just laid out that world in the first film. When I read the first treatment, there was so much depth of understanding of that world, and how everything in the world worked to give you, visually, what you saw, which is really a layer on top of all the structure that he had created. But I'm just really looking forward to the opportunity to go back to that, and to understand it better and to dig deeper, especially with the tools that we can now bring to it 10 years later.

That's what I was curious about: that was such a stunning achievement then, and here it's almost a decade later you're going to get started again. There must be so many things that you know you can do even better than you did the first time around.

Well, there's things that we've learned from it that we've just tried to bring to bear. Like when we did the first film, when we created that jungle, we took the approach that you would do like when you're doing a set, where the trees were art directed. We picked the trees that we liked -- Jim knew what he wanted for each one, and we picked them and designed them and built them, just like an art department would do.

But then we came up with this idea that in order to figure out how to lay them out, and we didn't want that to be arbitrary, we did this idea of competition for resources, right? So the big trees would crowd out the little ones -- big trees can only grow so far together -- and it gave it sort of a naturalistic feel. But if you look closely, these were trees that were all made by hand, for better or for worse.

After we learned that, we thought, "OK, well that competition for resources happens not actually between trees, but as the tree is growing." That's why every tree is unique. Branches die and shed. Wouldn't it be great if we could figure out how these plants actually grow? So we've been working on it ever since Avatar, just really out of curiosity.

And we had a chance to actually put that into play in War, because we had to create the mountain region around the hidden fortress. And it was this big pine forest that eventually we were able to destroy with an avalanche. And so we had this idea just clicking along for years and years, and then Matt said, "OK, I need to create this forest and have you destroy it." And we thought, "Wow, we can do this now."

So we pulled that out, and it was a piece of software we called Totara, which is the name of a New Zealand pine tree. And we seed the forest now and we grow the forest, and each tree grows uniquely. You basically just run this algorithm for 100 or 200 years basically, and you grow the forest. And then, we wipe it out with an avalanche!

So it's like, all these things, one thing leads to another, but the spark for that came from Avatar originally, and now we were able to just roll it out for Apes. But that's what you do. That was the big breakthrough for Avatar. You couldn't think about doing one effect or the other effect, you had to think about anything that you needed to create in the world, and also how they would work together. And that's just become the basis for so much of what we do on any film. So yeah, that really helped us in Apes, just to have been thinking about that for the last 10 years.

Page 3: [valnet-url-page page=3 paginated=0 text='Reuniting with James Cameron, What Makes Weta Special']

Can you say, in general terms, if the new Avatar movies will go deeper into the world established, or will it blow the world out and make it a universe?

I can't say!

You knew that working with Jim again was always a possibility, to return to this world. Creatively, what got you energized about re-exploring Pandora?

The fact that when I read the first treatment for the first film, there was so much depth there that actually didn't make it to the screen, so stuff that had to get shed because of time and story. What Jim wrote for that first treatment really hinted at a lot of complexity to that world. And to me, that equates to realism, but it also creates to dramatic possibility. So I just knew there was a lot more out there to explore. That's what was really exciting to me. I like that world, and it's great to be able to go back to it.

avatar

What's fun about working with Jim Cameron for you?

He's totally dedicated, totally in the moment. With Jim, he wants to understand everything about what we're doing. As I said, when he wrote that first treatment, there was a lot of depth just to the biology and the ecology of the way everything worked together. So when you're creating that, Jim gets into the science of it. He gets into the physics, he gets into the biology, he gets into the anatomy -- everything has to work in a way that's believable for this planet. There are no shortcuts. And I think that pays off on screen, because you don't really have to suspend disbelief. It's just there, and it works. And Jim is just fantastic about that.

So from a visual effects breakdown, we all work that way. That's our modus operandi. So to have someone like Jim who just really is right there with you -- and in some cases ahead of you -- talking about how this stuff really needs to work under the hood, that's great for us.

Do you have a bucket list after all of your accomplishments in visual effects? Do you still have little check marks where you're like, "If I can get to that, if I can achieve this, that'll be one more level up."

It's all really story-driven. I like looking at all aspects of what it takes to create something in front of the camera, whether it's a muscle system, or it's fur dynamics, or how tides work or how the camera responds to light. All of this stuff is interesting, and then each story lets you know where you need to focus.

There's been a sort of democratization about visual effects, and you're starting to see it on much smaller-scale projects, but almost on a similar level of quality that Weta has been doing all these years.

That's always been true throughout the history of this industry. You develop a technique that's really hard, and people figure out how to do it and figure out how to package it up and make it more usable to more people. Remember when ILM first did the morphing technique for Willow? It wasn't that long after that that you could buy software off the shelf that would let you do that. So that's always been the nature of the business. It's really hard to crack it the first time, but once you crack it, everyone shares this information.

We publish papers on it in various journals and things, so it's not like a lot of this is closely held secret. Because it really does come down to things like the physics of it. Once you crack it, you're publishing it because you need to get to the next level and you want other researchers out there to be able to build on it to figure out where to go, because that's only to our advantage.

So there's a lot of sharing that actually goes on in this business. And yes, a lot of that will make its way out into tools that are easier for more people to use, no doubt about it. And it's great. We're seeing a lot of good work, because at some point we do get to go home at night and just relax, and it's great watching shows like this -- like Game of Thrones: we actually just did an episode for them on the last season. And it's great to be able to just watch a show like that and enjoy it, and have the effects fit in and just do the right thing, and they're able to create this whole episodic journey for you. Because I love the books as well. So yeah, it's kind of great that it works out that way.

Because you've always put an emphasis on the storytelling side of what you do, do you want to direct yourself?

No, not particularly. I like doing what I do. I like just creating this side of it, and making it all just come together. Yeah, that keeps me busy enough!

Who were the gold standards when you were starting out? Who were the visual effects pioneers that you really admired?

Boy. I mean, I'd probably have to say in those days when I was starting out, really it was the work at ILM that was setting the standard -- actually pre-ILM, really, Star Wars and that whole team came together to create that. That was pretty unique. Obviously there was 2001 before then, but 2001 was kind of a unique journey in a one-off kind of way, whereas when you started seeing a film like Star Wars, there was so much of this science fiction world that you'd been reading for years and years that just came to life in that film. George and his whole team there were really able to take that and make you feel like this is a whole universe you could walk around in and explore. And in fact, that's what they did.

So when I finally got to ILM, I got to work with Dennis Murren and some of the guys there and it was just outstanding to be able to do that. Terminator was another one ... Because when I got to ILM we worked on Terminator 2, and it was just mind-boggling to watch that happening behind the scenes and seeing that all really come together. Because I'd been really interested in computer graphics before then, but it wasn't until I started working with Dennis and some of the people there that I really started to understand the filmmaking aspect of it and how you apply the two together.

Let’s close out on your thoughts about your team in particular, and what sets it apart from others? What is the thing that Weta does a breed apart from everybody else?

There's a few things that I try to do uniquely with Weta. One is that I still try to maintain oversight of all of the projects so that there's a cross-pollinization of techniques. No film is shut out from any technique that we could use. Anything that we're driving for to create, we're doing it almost from the background, almost irrespective of the film, and then pulling on the bit that we need for any particular film. So we try to keep that idea that any film has access to the cutting edge of the work that we're doing. But also, that we are looking at it creatively in its entirety.

And we're able to do that because our team's been together since Lord of the Rings. We've all been working together, and when we started breaking out the shows like Avatar, the sequences were so big that we had supervisors on each sequence, and they were so varied that it became not much different than actually working on multiple films at the same time, and looking at what makes them work, but also what's unique about them and where the focus has to be. So we try to treat every project that's going through as part of the whole of the work that we're doing.

And we also try to make the place very filmmaker-friendly. My ideal on a show is to do what we did with Apes, or to do what we're doing with <em.Alita, which is for us to take on the whole show. We just want to be the team that works with the director and the production to provide the visual effects. I just think, to me, that's the best model.

I know that's difficult given logistics and the size, but still, that's kind of standard practice in the industry for the most part. You have one director of photography, you have one production designer. So to me, having one visual effects team that's really immersed in the film makes sense. So we try to make it a home for directors to work in, but again, making sure that if someone needs our focus, they can get it.