Introduction to Netflix Quality Control (QC)
Purpose:
This article provides an overview of the four different types of QC at Netflix. The aim is to provide our production and delivery partners with an understanding of our QC processes and expectations in order to create a consistent, standardized and relevant QC protocol that can be used throughout the entire production chain by our partners. This document also outlines common asset failures and provides further context into Backlot redelivery requests, which is a reflection of our commitment to providing a great customer experience.
Types of QC:
-
Production QC (Vendor Supplied)
- Responsibilities: Maintains integrity of the captured image and begins a consistent protocol for flagging issues.
-
QC Operations
- Responsibilities: Protects the customer experience for all content. Examines transcoded material on our internal web player.
-
Branded Content QC
- Responsibilities: Ensures the technical quality for Netflix Branded Content to protect the member experience by reporting any technical anomalies, which would negatively impact the accurate representation of the creative intent.
-
Localization QC
- Responsibilities: Enforces style guides & translation consistency. Examines source files in our internal tool, Subtitle Originator, with the goal of ensuring the best quality possible.
Production QC
QC starts at the on set, dailies and editorial production stages and is a crucial part of a proper 3-2-1 process. It allows to reassure production to wipe and reutilize the original media cards and gives peace of mind that a take or a scene match the expected creative and technical quality: errors identified at the dailies and on set stages, that pose a potential challenge to the integrity of the captured image, can be dealt with by the shooting production team as soon as possible.
For more information on the Production QC process please refer to the article: The Production QC Glossary.
COMMON PRODUCTION QC ERRORS:
- Crew in shot
- Boom in shot
- Lens flare
- Under/Overexposure
- Soft Focus
- Corrupted media
*For a full list of Production QC Issue Codes, please refer to the Production QC Glossary. Additionally, a CSV of the Production QC Glossary may be found here.
QC Operations
QC Operations identifies issues that would be disruptive to the streaming experience. The QC Operations team takes the approach of simulating the customer experience. Focusing on:
- Issues that would prevent consumption (e.g. the incorrect content being delivered)
- Issues that noticeably reduce the quality of the title (translation issues, typos, video artifacts)
- Maintain consideration for licensed content (varying styles)
Delivery & Ingest:
The delivery/ingest process for content going through QC Operations consists of three steps: Asset Upload, Auto QC/Inspection as a Service (IaaS), and Manual QC. The following flow chart highlights this process:
Types of QC Operations Inspections:
-
Spot QC:
- First 2 minutes
- 1min: Find the credits
- 1min: 50%
- 1min: 75%
- Last 2 minutes
- Consists of a Spot Check at:
- Performed for assets with lower predicted popularity
-
Full QC:
- Entire asset is reviewed
- Performed for Netflix Originals and high profile content
Common Catalog QC Errors:
| Video | Audio | Timed Text |
|
Burned In Subtitles |
||
|
Segment bumper/recap |
Censored Content |
|
|
Ghosting |
||
Branded Content QC
Netflix Branded Content QC (BQC) applies specifically and exclusively to Netflix Branded content (i.e. any content that includes Netflix Idents and presentation cards/credits anywhere in the world). BQC validates technical adherence to the Netflix Post Production Branded Delivery Specifications while also focusing on video and audio anomalies that would disrupt the member experience and/or interfere with the intended representation of the creative vision.
Branded Content QC is performed on the following longform content* asset types, and each respective asset type is reviewed in a “unity” environment that can accommodate native 1:1 playback of each respective asset type, whenever possible.
- Primary A/V - Interoperable Mastering Format (IMF)
- Printmaster Audio - channel-based or Atmos BWAV ADM (.WAV)
* This is exclusive of “supplemental” content (e.g. trailers, promos, artwork, etc.)
Branded Content QC utilizes the Netflix Branded Content QC Error Code Glossary to define the issues flagged and the potential severity structure for those issues within content. The Netflix QC Issue Severity Explanation provides additional guidance around how the three severity structures should be applied to issues flagged within our content.
Branded Content QC Partners
Netflix Branded Content QC is performed by an ecosystem of external Branded Content QC Partners, who have been vetted by the Netflix Branded Content QC Program task force (BQC Pod) and Production Technology & Operations (PT&O) team. These partners adhere to a specific Netflix Branded Content QC Service Level Agreement (SLA), Scope of Work (SOW) and any other additional support and contextual guidance provided by the BQC Pod at the program level or PT&O Specialists at an individual title level. The current list of partners included in the BQC Program are:
- 3rd i Digital (UCAN)
- Audiomaster Candiani (LATAM)
- Eikon Group (UCAN/EMEA)
- Imagica Entertainment Media Services (APAC)
- Iyuno Asia (APAC)
- Resillion US (UCAN)
- Sony PCL (APAC)
Fix Notes Process
Netflix Branded Content QC includes a fix notes process within our Asset QC Platform that allows our external Production partners and/or their picture and sound finishing vendors to review and provide context on any errors flagged during the BQC process. This includes indicating they intend to provide a correction in the form of a redelivery (i.e. “Production Will Fix”) OR signing off/approving a given error (i.e. “Creative Intent” or “Best Possible”). They also have the option to utilize comments to provide additional context around a fix note selection to Netflix or to request additional clarification from Netflix or their vendors around a particular error. These comments are optional. Only the fix note selection is a required part of the fix notes process.
Upon submitting their fix notes, the PT&O Specialist and/or Post Title Management assigned to a given title will review and complete the process. IF both parties agree with all fix note selections, they will complete the Asset QC request fix notes process. If fixes are indicated based on “Production Will Fix” selections, it will FAIL the current Asset QC Request and put the associated Backlot or Content Hub requests into redelivery. If all issues are signed off/approved, it will pass the Asset QC Request and accept the associated Backlot or Content Hub requests.
IF there are any sign offs that may require additional consideration, the PT&O Specialist and Post Title Management team will flag that back to the external Production partner and/or applicable finishing vendor. Post Title Management owns the final say on whether or not a given issue requires a fix, based on schedule, budget, and/or resource availability considerations. The PT&O Specialist serves as a technical consultant in this process to advise on the potential member impact of a given issue.
Localization QC
Localization QC qualifies translation quality, consistency and style guide conformance. This process involves a QC operator reviewing the timed text asset, implementing changes and categorizing the reasoning for the changes.
Localization QC is performed on the following asset types:
- Subtitles
- SDH/Closed Captions
Localization QC Workflow - Subtitle Originator
Timed text assets are QCed in our proprietary tool, Subtitle Originator. Timed texted events are lined up with waveforms to verify sync and QC is performed.
Fixes to a timed text file are made directly in the tool and changes are shared back to the author and/or Fulfillment Partner in the form of a Localization QC Report in Subtitle Originator (if fulfillment partner has access) and Backlot:
Subtitle Originator:
Clicking on the report icon will provide access to The Change Log. The Change Log details all fixes that the Localization QC operator made to this file and the error categories, as shown below:
Backlot:
Following this "Link to the Localization QC Report" will provide access to The Change Log. The Change Log details all fixes that the QC Operator made to this file, as shown below:
Localization QC - Metrics Evaluation:
Error rates for each asset type are defined as follows.
-
Subtitle error rate:
- Events with errors / total # of events
-
Language error rate:
- Events with translation errors / total # of events
-
Technical error rate:
- Events with technical errors / total # of events
-
Audio error rate:
- Number of audio flags / total run time
Common Localization QC Errors:
- SGP (spelling, grammar, and punctuation)
- Objective Translation
- Subjective Translation
- Timing to Audio
- Reading Speed