There’s little doubt about it – ​Game of Thrones ​is one of the biggest television series ever produced. Telling an expansive story that stretches from the Seven Kingdoms of Westeros to the lands beyond the Narrow Sea, HBO’s show is a large and impressive undertaking indeed, particularly so when you consider it was created on a small screen budget.

For Image Engine, working in the world of top­-tier Hollywood productions is just another day on the job. However, the high pressure deadlines and towering expectations surrounding HBO’s highly successful Game of Thrones was another challenge altogether.

Given its size and popularity, Image Engine was flattered to be asked to bring its skill set to the world of Stark, Baratheon, Lannister and Targaryen; specifically the starkly beautiful lands of the far North.

“This was my first project at Image Engine, and I was hugely excited to get started,” begins Mat Krentz, visual effects supervisor on the project. Krentz brought a great deal of ​Game of Thrones experience to the table at Image Engine, having worked on the show’s previous two seasons during his tenure at Look FX. “​Game of Thrones ​is very high-­end television, and the quality demanded goes up every season.

“Everyone at Image Engine was super excited to get their teeth stuck in,” beams Krentz. “The team had to work with a new kind of pipeline and push work through a lot faster than they normally would, but people were just so thrilled to be a part of this huge, global phenomenon. Everyone wanted to get started and produce their best work.”

Read on to find out how Krentz and his team streamlined the Image Engine pipeline to meet the demands of an episodic production schedule, while also breathing new life into George R.R. Martin’s imagination…

Winter is Coming

Image Engine was responsible for 205 shots across G​ame of Thrones ​season five, ranging from set extensions of the Night’s Watch’s northern headquarters, Castle Black, to the splattering of one unfortunate character’s eyeballs. However, the episodic schedule and tight deadlines meant that quick turnaround on every shot was key, and that meant a restructuring of the way the Image Engine crew approached its work.

“Normally when working on a large­-scale movie production we’ll front-­load a lot of what we do in 3D,” begins CG supervisor Mark Wendell. “For example, on a show with a huge shot count like Chappie​, we would spend more time and effort on things like rigs and models, ensuring that whatever we sent on to the compositors was already fairly polished.”

“However – and there’s no value judgement in this – we took a very different approach with Game of Thrones ​due to the fast turnaround required of the work. We tended to push work quickly through assets, lighting, look dev and animation, and left a large amount of the work to the compositing team.”

“This meant three things,” continues Wendell. “One, it allowed us to keep our costs low in the 3D departments, which in turn kept things within the stipulated budget. Two, it allowed us to get iterations out much more quickly, as we wouldn’t be waiting on an asset to get developed as long as we normally would, or have to re­render each time we re­sent something to production. And three, it allowed the compositors to take a much more active role in the final look of the project, which was a great opportunity for the 2D team.”

Indeed, the tight deadlines extended opportunities to Image Engine artists across the entire Game of Thrones ​project, not just on the compositing side. “The need to complete work quickly and efficiently meant that we took advantage of generalists much more than we would on a full film pipeline, where artists tend to focus singularly on one task,” explains Wendell. “On ​Game of Thrones​, our trackers and matchmovers also worked on things like layout and even animation. They really enjoyed it: it’s an extra bit of experience on top of what they usually do, so they get to build their skills in a lot of different areas. Also, this approach it helped us cut down on production time, so it was a win­-win situation.”

The Watchers on The Wall

Lying as far north as many Westerosi dare to go, and nestled against the 700 ­foot blockade of ice that forms the boundary of the lands Beyond the Wall, is the primary headquarters of the Night’s Watch: Castle Black. It’s a stark, cold and uncompromising location – or, as Tyrion Lannister once put it: “Surprisingly beautiful. In a brutal, horribly uncomfortable sort of way.”

It was up to the Image Engine team to bring this location to life in all its glory, which meant not only extending the on-­location set itself, but also establishing the look of the Wall that Castle Black defends, adding a sense of authenticity to its formidable, towering sheet of ice.

The on-­location set of Castle Black was a half-­built version of the fictionalised version, little more than two storeys high. Image Engine needed to take this set and turn it into the impressive yet decrepit multi­tiered structure witnessed in the aired episodes.

“The half-­set was built against a rock wall that’s spray-­painted white to make it look snowy, so that gave the production team the option of not having to do any set extension for low­-angle shots,” begins Wendell. “However, for wider or high­-angle shots we would extend the walls or the castle itself. Those shots ranged from background extensions in scenes where characters were in the foreground to the larger establishing shots. For those wider, more panoramic shots, we would need to extend the sides and upper areas of the castle to make sure it appeared to be a full, complete building.”

To go about creating these impressive extensions, Image Engine was supplied rough LIDAR scans of the Castle Black set. “production had used those scans to built out a simple geometry showing what they expected the castle to look like in terms of size and shape, so we had the extension pieces already built out in rough form,” says Krentz. “We input a little bit of work into those assets and polished them up, ready to go into the Image Engine pipeline. We would do that by modelling in Maya; painting our textures in MARI; using our open source toolkit, Gaffer, for look dev and lighting; and then 3Delight for rendering.”

While the set extensions were a straightforward affair, several shots that also had to incorporate snowfall were a little more tricky. Here, the plate photography contained falling flakes, and the Image Engine team needed to make sure this snow matched up with what was falling on the extensions they created.

The team also created the rickety, wooden elevator that takes members of the Night’s Watch from Castle Black to the top of the Wall – another asset that proved a challenge. “The elevator was another partially built set element, “ says Krentz. “We did an extension for it, but that proved a little bit tricky, as it was quite hard to line up – it didn’t quite match our geometry. There was a bit of back­and­forth on getting it just right, and figuring out how the elevator was going to look so it worked in the overall scene. And then, of course, we had to make sure that the ice behind the elevator looked believable too…”

A Song of Ice…

The 300 ft glacial barricade intended to keep the Wildings separate from the Seven Kingdoms is one of ​Game of Thrones ​most enduring images, seemingly larger and more imposing with every new season. For season five, it was up to Image Engine to give this thick sheet of ice a realistic sheen.

“The look of Castle Black was pretty much already set, but we directed the ice wall itself a lot on our end,” begins Wendell. “The thing about the ice wall is that its look has varied a little bit over the years. Production showed us work from past seasons and pointed out the references they wanted us to match to, but because we hadn’t done the ice wall here before, we had to use our own tool set to try and reproduce that look. That meant a fair amount of fun discovery for us: we did a lot of research, looking at glacial materials and huge towers of antarctic ice.

“One of the challenges we were faced with, however, was that for a quite a few shots inside the elevator that travels up the side of the Wall, we were seeing the icy materials very close up,” continues Wendell. “It was a lot closer than was planned for in the model that we received from production as a reference, so we had to do a lot of work building up extra resolution for the geometry close to the camera. We spent a good amount of time figuring out the right icy look using our materials. It was a really about pleasing compositions, rather than sticking with the physics of the situation exactly.

“Again, that was a really nice collaboration between our 3D and compositing teams: the 3D team would take the asset as far as it could – perhaps not quite as far as we would on a film project – and then let the compositing guys experiment with different passes and layers to come up with a final look,” concludes Wendell. “The final product on screen really is a true partnership between the 2D and 3D departments here.”

The Pointy End

Blood, sex and swords – that’s a pretty good way to summarise any ​Game of Thrones ​episode, and Image Engine was responsible for at least two of those elements throughout the course of season five.

“Creating gore – from stab wounds through to gouged­out eyes and more – comprised a large amount of work that we did on the season,” says Krentz of the gruesome effects. “The shots varied between 2D elements, where the compositors were just enhancing a blow from a weapon on the plate with some composited green screen blood, to 3D extensions of weapons. In the latter case actors would have half swords or other weapons, and we would track those and replace the missing parts of the weapon using the location of the weapon in the actor’s hand as a reference point. It would then appear as if the actor has been actually speared by the weapon.”

The 2D team would enhance these shots with thick splatters of crimson­red blood – something they happened to have a great deal of access to, according to Wendell: “production provided us with an entertainingly large library of green screen blood elements,” he laughs. “Some crew members had sat on a stage just spraying and splattering packets of blood in all directions. We had all kinds of shapes and arcs to play with, so it became an incredibly useful library for the compositors to sell any shot where a sword is striking somebody or a pike is piercing a torso. We didn’t do a lot of simulated blood at all – it mostly came from that client-­provided ‘blood library’.”

One element where this library came in particularly useful was the battle between ​The Sons of Harpy and Barristan Selmy in the alleys of Meereen​. “It was a huge sequence for us,” says Krentz. “While working on it, production decided that the scene would look better with blood all over the walls. That meant we had to roto all of the foreground characters out separately and start doing 3D tracks for all of the walls, adding in splatter and spray across the entire sequence as the fight progressed.”

From stabs and slashes to outright beheading: Image Engine’s work also included some of the more gruesome death scenes witnessed throughout the season. This included the execution-­style decapitation of Janos Slynt by Jon Snow.

“To create the head stump, we tracked the shoulders, the neck and the head of the actor, and got a good matchmove of what they were doing in the plate,” explains Krentz. “We then did some tricky comp stuff to get a clean plate of where the actor’s head would be, meaning we then had a headless actor. We then inserted a digital stump.”

However, in the case of Janos Slynt the 2D approach didn’t quite cut it. “For some of the shots, in deference to the schedule and budget, we tried to get away with as much as we could with simple 2D/2.5D approaches,” says Wendell. “But with the Janos Slynt execution, production wanted the head to roll forwards once it had been lopped off. We couldn’t do that in 2D with the live actor’s head element, so we replaced it with a 3D projected version. It ended up being a much more effective shot.”

You Win or You Die

Decapitations were one thing, the scooping out of eyeballs something else altogether. Image Engine was also asked to create an a rather eye­popping scene to rival that featuring season four’s Oberyn Martell. In this case, the eye popee was none other than sinister Kingsguard Meryn Trant.

The creation of the grisly sequence started with the creation of in­house concept art, which was signed off by the executives at HBO. “From there we went into assets and worked on the model, textures and look dev,” says Krentz. “With this specific asset we did go a bit further than we did with others, as it was such a key sequence – it needed to look really good.”

While the actor on set was wearing a pair of prosthetics that mimicked the look of wounded eyes, the prop was poking out, rather than sticking in. “To counter this, the asset that we created was essentially a little raccoon mask in CG that was composited on top of his face to cover the prosthetic eyes, but otherwise keep as much as possible of the actor’s forehead, cheeks and temples visible as possible,” says Krentz. “We could then just replace the minimum amount necessary with CG eye sockets. We created sockets with gore and blood inside, and matched these 3D assets to the actor’s face from different angles and different shots. We then tracked it to the actor’s face and got it all locked off on the face. We could then really sell the fact that his eyes have been poked out with a knife. It was pretty disgustingly bloody stuff, but certainly effective!”

For Krentz, the process wasn’t full of the most joyous memories: “When we started work on the scene production asked us to send them as many references as we could of people with knives stuck in their faces. That was not the most pleasant of research jobs…”

The Others

From the grandiose to the subtle, Image Engine’s work on ​Game of Thrones ​also included 2D fire enhancement and additions to the weather. “There are a couple of sequences later on in the season where there needed to be a full­on snowstorm,” says Krentz. “A lot of that wasn’t shot in camera. So, we ended up having to roto out all of the characters in most of the shots and layer up snow and fog to give them more depth and atmosphere.”

Furthermore, the team was responsible for comping in the hulking giants and dire wolves that feature throughout the show. “For those enlarged creatures, the production team shoots the original photography of the actors – or wolves – against green screen,” explains Krentz. “With the dire wolf, those shots were lock offs, so we got footage of the real wolf shot on a green screen, and it was relatively simple to comp that into the scene at the scale it needed to be.

“However, the giants were a little more tricky,” he continues. “Because the camera is moving on a lot of the shots the giants are involved in, they needed to shoot the green screen footage using motion control cameras. The actors would run through the shot on set with motion control, and then the production team would use those same settings for the camera during the green screen shoot. That way, when it comes to us, we get the same camera movement from one plate onto the next. it’s a cool trick that works really well, and makes the compositing process that little bit more straightforward.”

There was one creature, however, that required a little more work – the seeming leader of the blue­-eyed White Walkers: the Night’s King. The reveal of this character towards the end of episode eight needed to be treated with absolute reverence, given the impact it was likely to have on the audience watching at home.

“That shot was another nice marriage between 2D and 3D,” says Krentz. “​We were tasked with making the actor’s prosthetic skin turn to ice; so anywhere it appeared on his face, hands and shoulders we replaced with a CG ice texture. ​3D was able to put together a bunch of different layers that we could use in comp, because we actually had to use the base of the costume and the actor as a layer, so it doesn’t make sense to build everything out in 3D. We ended up blending different layers of ice for the final look – we did a lot of research for this, using ice cubes and looking at what happens to light at different depths.

“Again, in deference to the tight schedule, we only built the look dev and the various layers to the point where we could hand them off to comp to create that final look,” continues Krentz. “The artists would basically have one NUKE script that would template how to reassemble all of the different passes once they had been approved. So, the final look really wasn’t 3D alone or 2D alone, it was put together between both – in a sense it’s letting compositing finish what you would otherwise do in a shader set­up. And the nice thing about that is there is a ton of flexibility and iteration that we can get in front of production without having to re-­render.”

The ​Game of Thrones​ project was therefore not just a chance for Image Engine to prove its abilities on one of the most famous and revered shows ever to grace the small screen, but also an opportunity for the team to find new, unique and streamlined ways to work. “It was a ton of fun,” smiles Krentz. “We all finished more than ready to jump on to the next season!”