REMINDER: Context is key; this information should be taken as broad strokes on a subject that can vary based on creative intent. Please read “2D LED In-Camera VFX Field Guide Overview” for context on where the information presented here comes from and is intended for.
Table of Contents
What Is In-Camera VFX?
With In-Camera VFX, the ultimate goal is to utilize display surfaces (such as LED walls with accurately projected content) to achieve the final visual effect in-camera during the shoot, requiring minimal to no post-production work.
Whether it's two people sitting in a car, a spaceship jettisoning into warp-speed, or recreating on-location lighting in a sound stage, interactive lighting technologies can empower cinematographers and their crews to create dynamic lighting and natural reflections on-set. This enhances the realism of the imagery and helps provide important creative cues.
There are two approaches for In-Camera VFX when it comes to display content: 2D or 3D.
2D In-Camera VFX |
3D In-Camera VFX |
|
|
2D In-Camera VFX involves projecting pre-rendered or live-action footage at a fixed perspective against a display surface. This achieves a high level of image quality and immersion. With driving comps, for example, plate arrays are shot at the same time and captured at a variety of angles. They are then reprojected on a display surface (LED screen) to emulate the experience of a sequence shot with a car process trailer; part of the value of this method is that it is easily repeatable on a controlled sound stage.
3D In-Camera VFX involves the use of image output from real-time engines to a live display surface, synchronized with camera tracking to produce final-pixel imagery in-camera. Content generated for 3D In-Camera VFX is not rendered into a final video format, but rather something that can be manipulated live through platforms such as Unreal Engine, Unity, or Notch (to name a few).
The Challenge and Possible Solutions
When watching content, you’ll often see talent in a vehicle, whether it be a car, plane, train, or boat. To achieve the shots in these traveling sequences, there are three shoot methodologies to choose from: practically on location, in front of a green or blue screen, or on a stage with LED panels.
On-Location Car Process Shoots
Traditional driving photography requires large process trailers to support the vehicle, camera car, and crew members. Permits and police officers are required to hold or direct traffic. Continuity might be sacrificed when the actors’ hair, makeup, and wardrobe support crew aren’t able to get to the performers due to time constraints or location logistics. Creative choice can be restricted by the changing light and the time it takes to reset the process trailer.
|
Example of a process trailer |
Example of a process trailer |
Green Screen Car Shoots
This method seems simple: load a vehicle onto a sound stage, roll up a green screen, and shoot! There are three key challenges with this, however. First, the car must be accurately lit as if it is traveling down a street, in a city, or out in the countryside at multiple times of day. Second, the Director must explain to the talent and editorial what is going on beyond the green screen. Third, the actual visual effects work requires an artist to key out the green screen, composite a practical driving plate seamlessly into the practical shot (or plate), and add reflections to the vehicle and talent. While logistically the simplest method, the end result may vary in quality.
Example of green screen car shoot.
To help with lighting and reflections on the car, LED panels or traditional rear/front-projection can be used for interactive lighting on set. This technique of pairing interactive lighting with green or blue screen backgrounds can eliminate some VFX work in Post.
Example of interactive lighting on sound stage.
2D In-Camera VFX
This methodology removes all the logistical challenges of traditional process trailer shoots as well as the artifacts associated with driving comps. The LED panels, serving as a background or window exterior, display moving images made up of practically photographed plates, wholly digital environments, or some combination of the two. The need to travel a large cast and crew to the location or to book a second unit in another country is eliminated. Depending on the scene, the number of crew members required to shoot on a soundstage may also be reduced.
Compared to traditional green screen composites, the use of In-Camera VFX also has the potential to reduce the resources required in Post Production to finish shots. By adding backgrounds to scenes in a studio environment with LED panels, the camera operator can frame as they would any real object, and the actors can react to the actual final imagery, live. This not only provides cast and crew with a visual environment to reference in real time, but all of the natural reflections and lighting from the screen provide important artistic cues and enhance the realism of the imagery - avoiding the typical struggles of a green screen (like unwanted reflections and color spill on the subjects).
But this solution is not without its own challenges - as the LED infrastructure can be cost prohibitive. However, there are a range of financial models to explore — from pop-up LED stages to purpose-built infrastructure designed for this type of work. The best execution of this work can result in the opportunity for savings on physical production costs, reduction in VFX shots, and an expedited post timeline - but more importantly, a more integrated visual look.
Visual Examples
Cars
Example of sound stage set up for in-camera VFX. Source: Territory Studios
Example of plate-based driving shoot.
Trains
Example of dailies from stage set up below.
Overhead visual of the soundstage for clip above.
Windows
Example of LED outside of the window.
Crew
Working with 2D In-Camera VFX will require a shift of some traditional crew responsibilities, as well as the addition of new roles.
The following is an example of roles and responsibilities for crew involved with 2D LED ICVFX workflows for car work:
ROLE |
RESPONSIBILITIES |
Director of Photography |
Critical to approve/lock in looks, more than on a traditional shoot, as the LEDs provide both lighting and what is being captured through the lens. |
Editorial |
Makes plate selects prior to the shoot and to assemble previs/techvis animatic for the Media Server Operator and additional crew so that they understand the number of setups and camera angles required. |
Production Supervisor |
Connects additional personnel for essential conversations and coordinates additional/alternative logistics (ie. connecting Transportation with LED Technician to understand what the layout of the walls will be, and figure out how to load in cars...) |
Production Designer |
Understands the scope of set build required ー what will be practical, and what will be displayed on the wall. Can help hide the blend where stage meets the bottom of the wall, through set design. |
Gaffer |
Assumes the responsibility of LED panel oversight and plays a critical role in ensuring safety and functionality with other lighting elements. |
VFX Supervisor |
If there are plans to incorporate VFX into display content, or onto plates in Post, it is imperative to have the VFX Supervisor involved in the pre-production process and on-set to avoid costly redos down the line. |
Post Supervisor |
Works with the Director/DP/Editorial to make plate selects for each scene, ensuring that they are up to spec according to the Virtual Production Supervisor and Media Server Operator. |
DIT |
Plays a crucial role in linking the creative look with the technical appearance of the LED panels. Helpful in identifying issues and can often aid with signal chain issues as well. |
| Virtual Production Supervisor |
A new and emerging role, it is similar to a VFX Supervisor and serves as the informed captain for all of the moving parts within the Virtual Production pipeline, ensuring that all infrastructure, teams, and phases of production are moving according to plan from prep through shoot. Works closely with the VFX, Post Supervisor and Production leads to ensure alignment throughout. |
Media Server Operator (or ‘Engine Operator’ in 3D) |
A new role that is responsible for the ingestion and operation of content on set and stages content according to shoot schedule. Also operates the display of content and positions/adjusts plates on the display, ultimately playing/resetting footage as needed. Depending on the infrastructure and scope of the shoot, two operators may be needed. |
LED Technician (or ‘LED Engineer’) |
A new position in charge of the physical set up, movement, adjustment, and tear down of LED wall infrastructure. Also necessary to have them on set during the shoot in case of any troubleshooting needs with the LED panels. |
* Title variations could exist since these are 'new’ roles in film/tv production and still being defined
Timeline
Due to the inherent nature of the pre-production planning needed to execute the in-camera visual effect, the order of operations and average timelines will be affected. Certain production crew will need to come on-board earlier than normal. Some decisions that may typically be left to figure out in Post will also need to be decided prior to the shoot.
The timeline below is a generic representation of the order of operations when prepping and shooting in-camera VFX with LEDs and 2D playback for driving comps, focusing on the importance/goal of each step.
- SCOPE ASSESSMENT: This is where level-setting and expectations are set. The first step is understanding what is trying to be achieved: What are the sequences (action, mood, time of day)? What are the areas of focus (shooting into LED has some limitations)? Are special effects going to be used (smoke, rain)?
- KICKOFF: Relaying the scope to the rest of Production, this is where alignment is needed before moving on to the next steps. For the DoP, it would mean understanding the number of crew needed for Lighting and Grip. The Gaffer will likely ask who is going to be operating the board. Depending on content being displayed, Post, VFX, and VProd Supervisor will likely need to work together to source/create footage.
- CONTENT/TECHVIS/STAGE RECCE: Each of these steps will inform the other in some way. On-Set Infrastructure and Plate Capture & Conform will need to be in alignment in terms of specs and requirements so that content can be properly displayed to expectations. Techvis can lay out potential camera options based on the content or show exactly how things need to be staged in order to achieve a desired shot. The layout of the stage will be a key part of Techvis in determining where everything/everyone will physically need to be on the day, to ensure that everything will fit, and that shots/sequences are achievable. Techvis can primarily solve content orientation or camera moves (whichever is more imperative to start), but will ultimately help to inform both.
- PERMITS/WEIGHT APPROVALS: To ensure production safety and confirm the weight capacity of the sound stage, the Transportation team will need to provide the make/model/weights of cars and the Lighting/Grip + LED teams will need to submit weights for rigging.
- INGESTION/ORGANIZATION: This is when the Media Server Operator receives and stages plates/files according to the call sheet. Prior to ingest, submission guidelines should be shared with whomever is responsible for submitting plates to the Op in order to help with the process (ie. codec, file naming convention). In addition to the call sheet, a cheat-sheet should be created to highlight what the Op should know about the plates (ex. ‘intentionally flopped’ ‘no LUT’). Ideally, plates will be prepped and QC’ed according to shoot schedule, so that scene swap can be done swiftly.
- LOAD-IN: Physical load-in of gear. Standard.
- RUN-THROUGH (or ‘PRE-LIGHT’): Stage and test everything. Set up lighting, review and orient the plates on the display as needed for setups (note: setups can be ‘saved’ like a file, so they can be queued up quickly on the day). This step is crucial, otherwise time will be spent doing this during production, and compromises will likely be made for the sake of time, resulting in unplanned fixes in Post.
- SHOOT / REVIEW: Showtime! Plate adjustments such as scale, orientation, and slight perspective shift are achievable as needed. Get the shot, reset instantly, and ship to Editorial when done. This is what it’s all about!
- STRIKE/LOAD-OUT: Physical load-out of gear. Standard.
- FIXES (IF NEEDED): Inevitably there may be some artifacts, moire, or other minor fixes needed.