NOTE: As a complement to this article, this video presentation gives a good overview of why Data Management is essential.
TABLE OF CONTENTS
The redundancy of original camera files (OCF) and audio files throughout a production is crucial. Data loss can be a huge financial burden and creative loss.
There are multiple ways to backup and verify production assets. Below are our minimum expectations followed by best practices.
- Capture Device (e.g., camera card or sound card)
- RAID 5, 6, or 10 (or above) storage (RAID 0 is only acceptable for temporary transfer/shuttle drives, and not for backup purposes.)
- LTO (6, 7, 8 or 9 written in LTFS format)
- Hold at least three copies of all original camera files (OCF) and audio.
- Store the OCF and audio copies on at least two different types of media.
- Keep at least one of these backups in a different geographical location from the others.
* Production is responsible for ensuring 3:2:1 practice is maintained through final delivery to Netflix.
If considering cloud storage options, including Content Hub, please contact Netflix to discuss.
If using MHL, a single MHL should be at the root of every camera and sound roll. One MHL per roll.
- During the checksum process, a unique machine-readable checksum manifest (e.g., MHL) should be generated with every offload and accompany the files through all the transfers, including final delivery.
- All copies of original camera and audio files should be offloaded using checksum verification, either MD5 or xxHash64be. Checksum verification is built into most professional offload software.
- If using MHL, a single MHL should be at the root of every camera and sound roll. One MHL per roll.
MD5 and xxHash64be are checksum formats
MHL (Media Hash List) is a manifest format (i.e., listing of file names and checksum values of those files)
Visual inspection of camera files during offload:
Whether you’re shooting scripted or nonfiction content, the best time to check for any image or sound file irregularities is during the offload. The visual inspection should cover any possible major recording or data transfer issues and serves as a technical check.
- Example method: Scrubbing through the material to perform a spot check of all captured media.
- Whether you’re shooting scripted or nonfiction content, the best time to check for any image or sound file irregularities is during the offload. The visual inspection should cover any possible major recording or data transfer issues and serves as a technical check.
Scripted Fiction Full QC of camera files:
- Providing feedback to key production personnel to discuss a solution to a technical problem.
Minimizing possible problems in the future.
- The primary purpose of Full QC is to catch correctable technical issues where there is value in finding those issues up-front.
- Full QC is ideally done at real-time playback
- Full QC should be performed at a minimum image resolution of 3840x2160 to best identify visual artifacts (e.g., dead pixels, moire, etc.)
- Full QC should be done in a controlled environment like a lab. If color evaluation is part of the process the monitor should be professionally calibrated.
- Full QC should be done from a checksum-verified safety copy and not a camera mag
Scripted Fiction Editorial Rushes Sign-off
- Editorial has access to reports from multiple departments such as Script Supervisor, camera loader and the DIT, and can cross-reference all of these reports with the lab report to confirm they have received every piece of shot material and potentially flag image inconsistencies or possible problems when reviewing.
REPURPOSING CAMERA MEDIA
Along with ensuring you have enough copies of the production assets, there are other considerations before wiping camera cards for reuse:
Repurposing Camera Media should only be done after the following:
Ideally, we would expect Editorial to perform final clearance, but if the turnaround doesn't allow for this, the cross-report check should at least be done by the dailies lab.
- Visual inspection of all footage has been performed against camera reports and if working on a scripted show, script notes.
- Original camera footage and audio files comply with the 3:2:1 principle stated above.
- On a scripted show, both a Full QC and a visual evaluation of all OCF has been performed and all the footage has been checked and accounted for against all relevant reports
- Ideally, we would expect Editorial to perform final clearance, but if the turnaround doesn't allow for this, the cross-report check should at least be done by the dailies lab.
REPURPOSING TEMPORARY ONSET STORAGE
If OCF is held on temporary storage, these are some considerations before wiping anything from that storage:
- Repurposing onset and near set storage should only be done after:
- The visual inspection of original camera and audio files has been done and it resides on a minimum of three checksum-verified storage mediums, including the final archival delivery format.
Acceptable Final Archival Delivery Formats:
LTO tapes (LTO6, LTO7, LTO8, written in LTFS format)
Netflix's Content Hub (Pending Netflix Approval)
Checksum Management and Chain of Custody
- After the original checksum-verified copy with machine-readable checksum manifests (e.g., MHL) has been completed, any subsequent copies should be validated against the checksum values from the original copy to ensure chain of custody.
Offload Bandwidth Speed
- The speed of the camera cards.
- The speed (including the interface) of the card reader.
- The speed of all of the drives or storage in your data chain.
- (If using LTO as your final archival delivery) The speed of writing and verifying to the generation of LTO being used.
- (If using Netflix’s Content Hub for final archival delivery) The speed of internet bandwidth for uploading files.
- The throughput speed of the devices you’re copying to and reading from is critical; faster drives will allow you to turn around camera media and drives sooner. A single slow device in the data management chain adds avoidable bottlenecks, adds unnecessary overtime, delays media turnaround, and challenges the success of a 3:2:1 workflow.
- It is recommended that drives should not be slower than the source (e.g., camera card or camera reader).
- Consider throughput speed throughout the whole data management chain
Camera Card Count
- Hold enough cards to complete the expectations laid out in the “REPURPOSING CAMERA MEDIA” section. This usually means having enough cards for two days of shooting; however always have the 3:2:1 conversation with all stakeholders before confirming that you have enough camera cards.
- Added link to "Data Management - Why It's Important."
- Added LTO9
- Removed "RAID 5, 6, or 10 (or above) storage (Pending Netflix Approval)" from the "Acceptable Final Archival Delivery Formats:" section.
- Added examples of data loss to reinforce the importance of redundant copies.
- Removed the Content Hub option within the 3:2:1 Principle since it is an alternate solution that requires further discussion with Netflix.
- Added example method of visual inspection.
- Renamed “Visual QC” to “Full QC” to differentiate from Visual Inspection.
- Renamed “Editorial Proxies” to “Editorial Rushes”.