Introduction to Netflix Quality Control (QC)
This document provides an overview of the three different types of QC at Netflix. The aim is to provide our delivery partners with an understanding of our QC processes. This document also outlines common asset failures and provides further context to Backlot redelivery requests, which is a reflection of our commitment to providing a great customer experience.
Types of QC:
- QC Operations
- Responsibilities: Protects the customer experience for all content. Examines transcoded material on our internal web player.
- Branded Content QC
- Responsibilities: Ensures the best possible quality for Netflix Branded Content. Checks the technical quality and content consistency of Audio and Video assets.
- Localization QC
- Responsibilities: Enforces style guides & translation consistency. Examines source files in our internal tool, Subtitle Originator, with the goal of ensuring the best quality possible.
QC Operations identifies issues that would be disruptive to the streaming experience. The QC Operations team takes the approach of simulating the customer experience. Focusing on:
- Issues that would prevent consumption (e.g. the incorrect content being delivered)
- Issues that noticeably reduce the quality of the title (translation issues, typos, video artifacts)
- Maintain consideration for licensed content (varying styles)
Delivery & Ingest:
The delivery/ingest process for content going through QC Operations consists of three steps: Asset Upload, Auto QC/Inspection as a Service (IaaS), and Manual QC. The following flow chart highlights this process:
Types of QC Operations Inspections:
- Spot QC:
- First 2 minutes
- 1min: Find the credits
- 1min: 50%
- 1min: 75%
- Last 2 minutes
- Consists of a Spot Check at:
- Performed for assets with lower predicted popularity
- Full QC:
- Entire asset is reviewed
- Performed for Netflix Originals and high profile content
Common Catalog QC Errors:
Burned In Subtitles
Branded Content QC
Branded Content QC validates Source Asset quality against our Netflix Originals Delivery Specifications and checks for content consistency. Branded Content QC is performed exclusively on Netflix branded content. Branded Content QC operators perform multiple full linear reviews of assets and flag issues that are inconsistent in the context of the program.
Branded Content QC is performed on the following asset types in a unity environment (e.g. 4K is watched in 4K; HD is watched in HD; Atmos is monitored in Atmos Speaker array):
- Primary A/V IMF
- 5.1 Music and Effects Audio
- 5.1 Printmaster Audio
- Atmos Audio
Branded Content QC Issue Codes can be categorized as follows:
- Technical Issues: consisting of Framing Issues, Aspect Ratio Issues, Image Levels, Audio Loudness, Peak Levels, Formatting, and more.
- Content Issues: consisting of Visible Production Equipment & Crew, Dead Pixels, Dropped/Duplicate Frames, Audio Ticks/Pops, Burned In Subs, etc.
Ultimately, Branded Content QC checks and flags inconsistencies introduced during the production and post-production processes. Issues are checked in a full linear QC and in a frame-by-frame capacity if need be.
Netflix Branded Content is QCed by a vetted Netflix Branded Content QC Partner after delivery to Netflix. All error codes and reports are verified to validate the actionability of an issue. Actionable reports are sent back to the delivery and Post teams to get issues fixed.
Why Branded Content QC?
- Flag issues that could negatively impact the customer experience
- Flag issues that detract from the original creative intent. It is unlikely that a boom mic, tape mark, or crew member were intended to be in the shot.
- Ensure adherence to Netflix Originals Delivery Specifications and technical standards.
Common Branded Content QC Issues (not ranked):
- Pixel Error
- Animation Error
- Frame Edge Error
- Visible Production Equipment
- Dolby Vision Metadata issues
- VFX Errors
- Audio Tick/Pop
- Luminance Shift
- Effects Sync in M&E
- Missing Effects in M&E
*For a full list of Branded Content QC Issue Codes, please refer to the QC Issue Code section of Partner Help Center or this helpful document on the Fix Notes process
QC Reports - Issue Code Severity Ratings & Action Items:
- FYI - Something that we would like to draw attention to that is not necessarily an issue -Will not trigger a redelivery request - No action necessary, but fixes can be applied if desired.
- Issue - Subjective issues that may not be detected by the customer (subjective translation and potential pronunciation issues) OR objective issues with the technical aspects of a file that customers would notice (issues that would negatively impact quality and/or do not preserve creative intent) - Will trigger a redelivery request - Post team must review and indicate what action they will take on the issue.
- Blocker - Issues that render the file unable to be consumed or would reflect very poorly on Netflix (wrong language, missing dubs, timing not in sync, offensive/derogatory language) - Will trigger a redelivery request - Post or Delivery team must fix and redeliver before launch.
Localization QC qualifies translation quality, consistency and style guide conformance. This process involves a QC operator reviewing the timed text asset, implementing changes and categorizing the reasoning for the changes.
Localization QC is performed on the following asset types:
- SDH/Closed Captions
Localization QC Workflow - Subtitle Originator
Timed text assets are QCed in our proprietary tool, Subtitle Originator. Timed texted events are lined up with waveforms to verify sync and QC is performed.
Fixes to a timed text file are made directly in the tool and changes are shared back to the author and/or Fulfillment Partner in the form of a Localization QC Report in Subtitle Originator (if fulfillment partner has access) and Backlot:
Clicking on the report icon will provide access to The Change Log. The Change Log details all fixes that the Localization QC operator made to this file and the error categories, as shown below:
Following this "Link to the Localization QC Report" will provide access to The Change Log. The Change Log details all fixes that the QC Operator made to this file, as shown below:
Localization QC - Metrics Evaluation:
Error rates for each asset type are defined as follows.
- Subtitle error rate:
- Events with errors / total # of events
- Language error rate:
- Events with translation errors / total # of events
- Technical error rate:
- Events with technical errors / total # of events
- Audio error rate:
- Number of audio flags / total run time
Common Localization QC Errors:
- SGP (spelling, grammar, and punctuation)
- Objective Translation
- Subjective Translation
- Timing to Audio
- Reading Speed