Welcome to the Netflix Partner Help Center. Have a question or need help with an issue? Send us a ticket and we'll help you to a resolution.

Snowball Edge - User Guide & Delivery Workflow

Introduction
Prerequisites
     Active Backlot Account
     API Key
     Content Hub Asset Requirements
     Firewall Configuration
     Hardware Recommendations
Phase 1: Setting Up
     Connect Snowball to Infrastructure
     Download and Install Netflix Content Hub CLI (aka Netflix CH CLI)
     Clean Snowball Edge
     Checksum Manifest Generation
          Use Case #1
          Use Case #2
Phase 2: Transferring Data
     Copy Action
          Use Case #1
          Use Case #2
          Use Case #3
     Finalize Action
     Return Snowball Edge Device to AWS
Phase 3: Wrapping Up
     Submit Content to Requests in Content Hub
     Communicate with your Netflix Representative
Appendix: Netflix CH CLI Options

 

Introduction

This article aims to provide an overview of how to use utilize an AWS Snowball Edge device, in conjunction with the Netflix Content Hub CLI, for delivery of assets into Content Hub.

 

Prerequisites

Before you can utilize AWS Snowball Edge for delivering assets to Content Hub ensure you have addressed the following prerequisites.

Active Backlot Account

  • The account should belong to the user that will take receipt of the Snowball Edge device and will be performing the transferring of data.
  • The account needs to be added to the Content Hub project and given a role that allows the user to upload content.
  • The data transferred via Snowball will only appear in the designated users Workspace in Content Hub. Therefore if the wrong account is specified during the Snowball ordering phase then please submit a request via the Partner Help Center to update the recipients email address.

API Key

  • There is a 1:1 relationship between an API key and a facility. 
  • API keys are not user specific.
  • An API Key need only be generated once and securely stored for re-use. 
  • If you have not already generated one, review "How do I generate an API key?"
  • The values generated here are your “Client ID” and “Client Secret”.
    • Client ID: Will start with the word “backlot” and have 5 alphanumeric blocks separated with a hyphen (-).
    • Client Secret: An unbroken 64 character alphanumeric string.

Content Hub Asset Requirements

  • Ensure that you have reviewed the relevant "Content Hub - Preparing Assets" documentation prior to organising content for delivery to ensure compliance to the latest requirements.

Firewall Configuration

  • Please ensure the Mac workstation used for the file transfer can initiate outbound traffic to the following endpoints using the HTTPS protocol on port 443:
    • https://all.baggins.prod.netflix.net
    • https://productionsuite-ingest.prod.netflix.net
    • https://meechum.netflix.com

Hardware Recommendations

  • Mac Workstation 
    • CPU: 12~16 cores
    • RAM: 32GB
    • HDD: 50+ GB of available disk space
  • Content Storage Volume
    • Read speed > 500MBs/s
  • Network I/O Support
    • 10Gb Ethernet RJ45 /SFP+ or 40Gb Ethernet /QSFP+
  • Network Adapters & Cabling
    • 10G network card or Thunderbolt 2 /3 to 10G adapter, e.g. SanLink2/3
    • TWIN10G-TB2 Dual Port 10 Gigabit Ethernet Thunderbolt 2 Adapter
    • TB2 and ethernet Cat6 cable

Please Note: Each AWS Snowball Edge appliance might or might not ship country-specific power cables. No other cables or optics are provided.

Examples of some common setups can be found in the Snowball Edge - Hardware Configuration Examples.

  

Phase 1: Setting Up

Connect Snowball to Infrastructure

  • Open the front and back doors, sliding them inside the appliance door slots. Doing this gives you access to the touch screen on the LCD display embedded in the front of the appliance, and the power and network ports in the back.
  • Open the top door and remove the provided power cable from the cable nook, and plug the appliance into power.
  • Connect the Snowball Edge device as shown above using the appropriate cabling.
  • Power up the Snowball Edge device using the power button located above of the LCD display.
  • When the appliance is ready, the LCD display shows a short video while the appliance is getting ready to start.

Please Note: It can take between 10-15 minutes for the Snowball Edge device to be boot up and be ready for use.

Download and Install Netflix Content Hub CLI (aka Netflix CH CLI)

The Netflix CH CLI is a terminal application you can use to unlock Snowball Edge devices, enabling them to be used for data transfer. In addition, you may use it to calculate checksum, generate report and create Content Hub workspace snapshot of your local files and folders structures.

Download and extract the Netflix CH CLI to your local workstation.

Expand the archive file and save to any folder. Once expanded, the executable "chcli" can be found under "${installationDirectory}/bin/"

e.g. chcli-1.44.0/bin

Please Note: Do not use the “snowballEdge” executable under the “runtime” folder. This is the default AWS client and will not work with Content Hub.

Clean Snowball Edge

This step is recommended if the workstation has been used for a different job or the current job needs to be corrected with a different set of source files. 

This step removes temporary files in workstation and delete all files on snowball as well as Content Hub workspace snowball folder that is associated with the job. It should only be used to prepare the workstation for the next job, or reset to initial state for the current job.

Reset command:

./chcli -action clean -email [your email/username] -clientId [Your backlot clientId] -clientSecret [secret] -snowballIp [IP Address]

Or

./chcli -action clean

(You will be prompted for required information)

Checksum Manifest Generation

Certain asset types require a Checksum Manifest (also referred to as a Data Snapshot or checksum.txt file) to be created prior to delivery. Please review the relevant documentation related to the asset type(s) you are processing. This can be found on the Partner Help Center located under Specifications & Guides -> Content Hub.

If a Checksum Manifest is required the Netflix CH CLI can be used to generate it by running the following command:

./chcli -action checksum -dir [Path/to/SourceFolder]

The end result will be a checksum.txt file in each of the relevant directories.

Below are are two example use cases. While the use cases use episodic content as an example it can apply to any type of content.

Use Case #1: Content Is Available Immediately

All data is organized per the requirements outlined [insert article name], and it is intended to generate a Checksum Manifest (checksum.txt file) for all data that will be delivered via the Snowball Edge device in a single session.

Simply point to the desired top level directory that contains all the content. The Netflix CH CLI will then generate a Checksum Manifest for for the assets in sub-directory, recursively.

In the example below a TV Show consisting of 4 episodes is organised on a single storage volume. The folder “FakeExampleShow_s01” (highlighted red) is the source directory specified during the “checksum" action. Four checksums are then generated in the appropriate subdirectories (highlighted yellow).

Snowball_UseCase1_Checksum.png

 

Use Case #2: Content Becomes Available Over Time

It is not uncommon for various assets to be ready for delivery at different times for example a Video Display Master (VDM) may be available before a Non-Graded Archival Master (NAM). To continue the example outlined in Use Case #1

Following on from the example in Use Case #1, two new episodes have become available AFTER the initial “checksum” action has been executed. There are two options:

Option A: Rerun the checksum action by pointing to the original source directory. In this example that would be "FakeExampleShow_s01” (highlighted red). This will regenerate checksums for ALL the contents and overwrite all existing checksum.txt files (highlighted yellow).

Snowball_UseCase2a_Checksum.png

Option B: Run the checksum action on the NEW directories only (highlighted green) by executing a checksum action specifying each new folder as the source directory, one at a time. This will leave the original Checksum Manifests untouched (highlighted yellow), and only generate them for the new content (highlighted purple).

Snowball_UseCase2b_Checksum.png

 

Phase 2: Transferring Data

The Netflix CH CLI “copy” command copies files and folders from your data source to the Snowball. Further in this article a few example use cases are outlined. While no use case is content type specific i.e. feature vs. episodic, the example used outline the most likely scenarios.

Copy Action

The "chcli" copy command copies files and folders from your data source to the Snowball. When copying data, prepare a source path and IP address of the Snowball appliance, as well as security credentials mentioned above. Enter the copy command as the following:

./chcli -action copy -sourceDirectory [Path/to/SourceFolder] -email [your email/username] -clientId [Your backlot clientId] -clientSecret [secret] -snowballIp [IP Address]

Or

./chcli -action copy -dir [source]

(You will be prompted for required information)

When you have entered the above command line the following will occur:

  • You will be prompted to enter a password matching your email | username.
    • These details relate to the user that was specified during the prerequisite phase.
  • You will be prompted to select a project (number).
    • If you are working on multiple projects you will see all of them listed here.
    • Take care to ensure you select the correct project for the Snowball being used.
  • You will be prompted to enter a Job Name e.g. NFLX-QXPW9599.
    • This is communicated via the Snowball Status Update emails that are sent to the user that was specified during the prerequisite phase. 

The CLI will look similar to the following:

Snowball_CLI_PID_TID_01.png

Once you have entered the above information the Netflix CH CLI will automatically:

  • Scan and index your source directory
  • As the transfer progresses a Snowball folder within the uploaders Workspace in Content Hub.
    • The Snowball folder (highlighted red) will retain the folder structure of the source directory specified during the “copy” action (highlighted yellow) and add place folder entries for the assets (highlighted green)
    • A blue spinning icon is used to indicate the material is still “in transit” to AWS. Once successfully ingested into AWS (after device is shipped back) the blue spinning icon will disappear.
    • WARNING: Do not attempt to use the “Move” or “Delete” functionality within the Content Hub UI while the data is still transferring and the blue spinning icon is visible.

Snowball_WorkspaceExample.png

  • Calculate a checksum for every file in the provided source folder and look for previously generated checksum.txt file (See Checksum Manifest Generation) to compare and verify checksum values.
  • Initiate the data transfer to the Snowball Edge device.

The following script can be used to introduce a degree of automation by queuing a "checksum" and "copy" action:

./chcli -a checksum -dir ${Your Source Directory} && ./chcli -action copy -dir ${Your Source Directory} -email $YourEmail -p '${Your Password}' -pid ${The Project ID} -tid ${trackingId}

The "-pid" is the Project ID that can be found in the Content Hub URL for the project.

E.g. https://contenthub.netflix.com/projects/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/requests

In a real world project the x’s will be replaced with the actual Project ID.

The “-tid” flag corresponds to the "Job Name" that is communicated in the Snowball Status Update email.

Use Case #1: Content Resides On Single Storage Volume & Is Available Immediately

All data is organized per the requirements outlined [insert article name], and it is intended to copy all data to the Snowball Edge device in a single session.

Use Case #2: Content Resides On Single Storage Volume & Becomes Available Over Time

It is not uncommon for various assets to be ready for delivery at different times for example a Video Display Master (VDM) may be available before a Non-Graded Archival Master (NAM). In such cases:

  • Execute an initial “copy” action on a top level folder.
  • As new material becomes available simply add the new folders anywhere under the top level folder.
  • When material is in place simply rerun the original “copy” action.

In the example below it can be seen that Session 1 was run when the VDM was available. When the Graded Archival Master (GAM) became available Session 2 was performed. Later when the NAM was available Session 3 was performed.

Snowball_UseCase2_CopyAction.png

For each session the folder “Bright” was specified as the source directory. The Netflix CH CLI keeps track of the data it transfers and is able to identify what material in the source directory already exists on the Snowball. It will then proceed to copy across only the new assets.

Please Note: Certain assets require a Checksum Manifest to be created prior to transferring to Snowball. Please refer back to Checksum Manifest Generation for more information.

Use Case #3: Content Split Across Multiple Storage Volumes & Becomes Available Over Time

It is not uncommon for content to reside on different storage volumes and become available over time rather than all at once. In the example below there are 4 episodes split across two storage volumes:

Snowball_UseCase3_02.png

As the Netflix CH CLI is able to utilise symbolic links and decipher subdirectories it is possible to run a single “copy” action instead of running individual “copy” actions for each top level folder spread across multiple storage volumes. To do this:

  • Create a temporary folder e.g. “Snowball”.
  • Within the “Snowball” folder create symbolic links to the desired top level folders.

This will result in a directory that will be similar to the following:

Snowball_UseCase3_CopyAction.png

Now it is possible to use the “Snowball” folder (highlighted red) as the source directory for the “copy” command”.

As more content becomes available over time additional symbolic links can be created and added to the temporary “Snowball” folder and the exact same command can be rerun when the content is ready to be copied.

The Netflix CH CLI keeps track of the data it transfers and is able to identify what material in the source directory already exists on the Snowball. It will then proceed to copy across only the new assets for each session.

In the example below it can be seen that Session 1 addressed the original material that was ready previously, in this case episodes 1 to 4.

For Session 2 new symbolic links for episodes 5 and 6 were added and the copy command was rerun on the "Snowball" folder. Only episodes 5 and 6 were copied during that session.

For Session 3 the same was done for episodes 7 and 8.

Snowball_UseCase3_01.png

Please Note: Certain assets require a Checksum Manifest to be created prior to transferring to Snowball. Please refer back to Checksum Manifest Generation for more information. 

Finalize Action

The "chcli" finalize command validates all the transfer statuses for the objects on the Snowball Edge by verifying the checksums computed before and after the file is transferred. If mismatches are found, a report will be displayed. Upon receiving a success response, the Snowball device is ready to be shut down and prepared for shipping back to AWS datacenter.

Finalize Command:

./chcli -action finalize -sourceDirectory [Path/to/SourceFolder] -email [your email/username] -clientId [Your backlot clientId] -clientSecret [secret] -snowballIp [IP Address]

Or

./chcli -action finalize -dir [path/to/source folder]

(You will be prompted for required information)

Return Snowball Edge Device to AWS

Congratulations! Now you can ship the snowball back to AWS:

  • Make sure that you've finished transferring all the data for this job to or from the AWS Snowball Edge appliance.
  • Press the power button above the LCD display. It takes about 20 seconds for the appliance to power off.
    • Note: A shipping label should appear after about a minute on the E Ink display on top of the appliance.
  • Disconnect and stow the power cable the AWS Snowball Edge appliance was sent with in the cable nook on top of the appliance.
  • Close the three doors on the back, the top, and the front of the AWS Snowball Edge appliance, pressing in on each one at a time until you hear and feel them click.
  • Arrange for the pickup of the AWS Snowball Edge via courier.
    • The courier information will appear on the E-Ink display with the exception of those shipped in Japan. In this case the information is attached to the side of the box.
    • Note: Ensure your courier understands that the E-Ink display on the top of the appliance should remain free of any and all adhesive labels.

 

Phase 3: Wrapping Up

Submit Content to Requests in Content Hub

Once the Snowball Edge device has been shipped back to AWS the content will be ingested and made available in Content Hub via the users Workspace. At this point the content is only visible to the user who performed the upload. In order for Netflix to take receipt of the material the content will need to be submitted to their corresponding request.

For more information on how to submit content to a request from a users Workspace please review Content Hub Overview - Assets - Workspaces.

For more information on how to monitor the status of a delivery after submitting the content via please review Content Hub Overview - Requests - Monitoring & Submitting.

If after reviewing the Content Hub documentation you still have questions please reach out to your Netflix representative for further assistance.

Communicate with your Netflix Representative

After delivery please be sure to communicate with your Netflix representative prior to deleting any content from your system.

 

Appendix: Netflix CH CLI Options

A full list of options available within the command line tool can be found within Snowball Edge - Netflix CH CLI Options.


Change Log:

2018-10-22

  • Phase 2: Added note to explicitly state that a partner should refrain from using the newly release "Move" or "Delete" functionality in Workspaces while data is still transferring to Content Hub (as indicated by presence of blue spinning icon).
  • Phase 2: Updated explanation of how to identify the correct value for the -pid flag when used in the outlined script.

2018-08-15

  • General: Article title has been update from " Snowball Edge - Quick Start User Guide" to " Snowball Edge - User Guide & Delivery Workflow".
  • General: The "Netflix Snowball Client" has been renamed to the "Netflix Content Hub CLI" and is also referred to as "Netflix CH CLI" in documentation.
  • General: Executable was previously called "snowballedge". It has been renamed as "chcli" to avoid being confused with the default "snowballedge" executable provided by AWS. All command examples have been updated to reflect this name change.
  • General: Reformatted "Steps" into "Phases" and added images to provide visual aids.
  • Prerequisite: Provided additional context regarding "Active Backlot account".
  • Prerequisite: Provided additional context regarding "API Key".
  • Prerequisite: Individual IP addresses removed and replaced with https://productionsuite-ingest.prod.netflix.net which needs to be whitelisted.
  • Prerequisite: "Hardware Recommendations" updated to provide clarity.
  • Phase 1: Added bullet point to ensure user is using correct/intended executable.
  • Phase 2: Added example script that can be used to queue a "checksum" and "copy" action.
  • Phase 3: Provided additional context regarding the actions a partner must take after returning the Snowball Edge device.
Was this article helpful?
2 out of 3 found this helpful