Introduction to Netflix Quality Control (QC)
This document provides an overview of the three different types of QC at Netflix. The aim is to provide our delivery partners with an understanding of our QC processes. This document also outlines common asset failures and provides further context to Backlot redelivery requests, which is a reflection of our commitment to providing a great customer experience.
Types of QC:
- Catalog QC
- Responsibilities: Protects the customer experience for all content. Examines transcoded material on our internal web player.
- Master QC
- Responsibilities: Ensures the best possible quality for Netflix Originals. Checks the technical quality of A/V and audio assets. Examines source files on QC vendors’ systems.
- Localization QC
- Responsibilities: Enforces style guides & translation consistency. Examines source files in our internal tool, Subtitle Originator, with the goal of ensuring the best quality possible.
Catalog QC identifies issues that would be disruptive to the streaming experience. The Catalog QC team takes the approach of simulating the customer experience. Focusing on:
- Issues that would prevent consumption (e.g. the incorrect content being delivered)
- Issues that noticeably reduce the quality of the title (translation issues, typos, video artifacts)
- Maintain consideration for licensed content (varying styles)
Delivery & Ingest:
The delivery/ingest process for Licensed Content going through Catalog QC consists of three steps: Asset Upload, Auto QC/Inspection as a Service (IaaS), and Manual QC. The following flow chart highlights this process:
Types of Catalog QC:
- Spot QC:
- Consists of a Spot Check at:
- First 2 minutes
- 1min: Find the credits
- 1min: 50%
- 1min: 75%
- Last 2 minutes
- Consists of a Spot Check at:
- Full QC:
- Entire asset is reviewed
- Performed for Netflix Originals and high profile content (at Netflix’s discretion)
Common Catalog QC Errors:
Burned In Subtitles
Master QC validates Source Asset quality against our Netflix Originals Delivery Specifications and checks for content consistency. Master QC is performed exclusively on Netflix Original content. Master QC operators perform multiple full linear reviews of source assets, and flag issues that are inconsistent in the context of the program itself.
Master QC is performed on the following asset types in a unity environment (e.g. 4K is watched in 4K; HD is watched in HD):
- Primary A/V IMF/ProRes
- Secondary Dubbed Audio
Master QC Error Codes can be categorized as follows:
- Technical Specifications: consisting of Framing Issues, Aspect Ratio Issues, Image Levels, Audio Loudness, Peak Levels, Formatting, and more.
- Content Issues: consisting of Visible Production Equipment & Crew, Dead Pixels, Dropped/Duplicate Frames, Audio Ticks/Pops, Burned In Subs, etc.
Ultimately, Master QC checks and corrects inconsistencies introduced during the production and post-production processes. Issues are checked in a full linear QC and in a frame-by-frame capacity if need be.
The Majority of Netflix Originals are QCed by a Netflix Preferred Vendor. All error codes and reports are verified internally, validating the actionability of a flag. Actionable reports are rejected back to both Fulfillment and Content Partners.
Using a Master QC NPV prior to delivery increases the likelihood of QC approval at Netflix.
Established rates are available on our NPV Website.
Why Master QC?
- Flag and correct issues that could negatively impact the customer experience
- Flag and correct issues that detract from the original creative intent. It is unlikely that a boom mic, tape mark, or crew member were intended to be in the shot.
- Ensure adherence to Netflix Originals Delivery Specifications and technical standards.
Common Master QC Errors:
- Pixel Error/Stain
- Animation Error
- Frame Error
- Visible Production Equipment
- Compression Artifacts
- Compositing Errors
- Audio Tick
- Luminance Shift
- Ident - Peak Levels Low
QC Reports - Error Code Severity Ratings & Action Items:
- FYI - Something that we would like to draw attention to that is not necessarily an issue -Will not trigger a redelivery request - No action necessary.
- Needs Review. Subjective issues that may not be detected by the customer (subjective translation and potential pronunciation issues) - Will trigger a redelivery request - Studio is expected to review, but we trust the studio to exercise their best judgment in addressing. Netflix will not review or approve any studio responses.
- Needs Fix - Objective issues with the translation or technical aspects of a file that customers would notice (incorrect Key Names & Places "KNP" terms, objective translation issues that would impact customer experience, and other issues that would negatively impact quality and/or do not preserve creative intent) - Will trigger a redelivery request - Studio/NPV must implement fixes and redeliver.
- Blocker - Issues that render the file unable to be consumed or would reflect very poorly on Netflix (wrong language, missing dubs, timing not in sync, offensive/derogatory language) - Will trigger a redelivery request - Studio/NPV must fix and redeliver in priority, before launch.
Localization QC qualifies translation quality, consistency and style guide conformance. This process involves a QC operator reviewing the timed text asset, implementing changes and categorizing the reasoning for the changes.
Localization QC is performed on the following asset types:
- SDH/Closed Captions
Localization QC Workflow - Subtitle Originator
Timed text assets are QCed in our proprietary tool, Subtitle Originator. Timed texted events are lined up with waveforms to verify sync and QC is performed.
Fixes to a timed text file are made directly in the tool and changes are shared back to the author and/or Fulfillment Partner in the form of a Localization QC Report in Subtitle Originator (if fulfillment partner has access) and Backlot:
Clicking on the report icon will provide access to The Change Log. The Change Log details all fixes that the Localization QC operator made to this file and the error categories, as shown below:
Following this "Link to the Localization QC Report" will provide access to The Change Log. The Change Log details all fixes that the QC Operator made to this file, as shown below:
Localization QC - Metrics Evaluation:
Error rates for each asset type are defined as follows.
- Subtitle error rate:
- Events with errors / total # of events
- Language error rate:
- Events with translation errors / total # of events
- Technical error rate:
- Events with technical errors / total # of events
- Audio error rate:
- Number of audio flags / total run time
Common Localization QC Errors:
- SGP (spelling, grammar, and punctuation)
- Objective Translation
- Subjective Translation
- Timing to Audio
- Reading Speed