In the last post, we discussed the various cloud native ways of building and running your VOD transcoding workflow. In this post, we will talk about why separating the encoding and packaging workflows is a good practice for VOD delivery.
In today’s world, every streaming platform needs to support multiple playout formats. This is largely because of the number of different playout devices and custom specifications for playing videos.
To deliver content to the end user in the most compatible format, we do two things: A) Encoding — To reduce the overall video size using some popular codecs like H.264, H.265 and VP9; B) Packaging — Covert the encoded video assert into a consumable streaming format like HLS or DASH and also enable DRM
Traditional Approach
1. Encode and pre-package each ABR format (HLS and DASH) separately across multiple renditions
2. Upload all the files to an Object Storage like GCS or some origin server for delivery
3. Pay a licensing fee for the number of ABR output formats
4. Pay an infra/storage fee for the additional storage consumed
Dynamic Packaging Approach
With dynamic package a.k.a just in time packaging a.k.a on the fly packaging, you go by this thumb rule — Encode Once, Package on the fly.
- Encoding — Encode your source content across multiple renditions and upload it to a source location (E.g. Google Cloud Storage)
- Dynamic Packaging — Detect the playback capabilities of the client and dynamically generate the necessary ABR format. For e.g. For a HLS client, generate the necessary .m3u8 and ts files
In this article, we’ll make a case for the following
With dynamic packaging, you can significantly reduce net encoding and storage costs and also save a lot of time from re-encoding and re-packaging your existing content library to support newer formats
How does this work?
Before we get into the details of the architecture, let’s take a moment to thank and appreciate the ever amazing open source community. Their contribution to the video streaming space is extremely valuable. In no specific order,
Kaltura’s nginx vod module — Enables On-the-fly repackaging of MP4 files to DASH, HDS, HLS, MSS. There are several other features like DRM, multi-bitrate , subtitles etc. supported by this module
Fluent-ffmpeg Nodejs module — This library abstracts the complex command-line usage of ffmpeg into a fluent, easy to use node.js module. Ensure you have ffmpeg and all other necessary encoding libraries like libmp3lame, libx264 etc. installed in your machine or docker container.
Now, back to the architecture. Here’s what the flow looks like
- Encode your mezzanine video files using the nodejs-ffmpeg encoder on Google App Engine
- Store the encoded assets in a GCS bucket location
- Configure your nginx-vod-module to fetch the video files from this location dynamically for HLS and DASH
- Dockerize your nginx-vod-module and deploy it on App Engine
- Set the App Engine URL endpoint as the origin for Cloud CDN
- Test Playback from a player of your choice — VLC, Safari, Bitmovin Player etc.
GitHub Repo Link for the Encoder Engine
saibalaji4/diy-nodejs-encoder
A fluent-ffmpeg based nodejs encoder. Contribute to saibalaji4/diy-nodejs-encoder development by creating an account on…github.com
GitHub Repo Link for the basic dynamic packaging module
saibalaji4/diy-jit-packager
This is a just in time packaging module using kaltura's open source nginx-vod-module GitHub is home to over 50 million…github.com
Performance considerations
- Packaging is basically a function of changing the container format and possibly adding DRM. This is usually a lightweight operation
- When it comes to massive streaming in the order of several million requests (think live sports), we may run into scaling/performance issues
- Dynamic packaging is usually a good fit for several thousands simultaneous users. With cloud native ancillary solutions like containers and Kubernetes, this can be further scaled in several orders of magnitude
Choosing the right solution strategy
There are solution approaches to do dynamic packaging and Open Source is not the only way or always the best approach. Depending upon your comfort level and engineering appetite, you can go with one of the below options
Vendor — SaaS based packaging solution
- The infrastructure and platform elements are completely abstracted from the user and managed by the vendor
- Usually comes with a portal where you configure the packaging settings
- Mark up costs — Usually bundles infrastructure and storage with the packaging service costs
Net Cost = SaaS Bundle Cost + Support/ Services Cost
Vendor — License based packaging solution
- The vendor provides the software as a deployable SDK that can be deployed on any hosting platform — Cloud, on-prem etc.
- Some vendors support a cloud native deployment model — container, kubernetes etc. on top of VM based deployment model
Net Cost = License cost + Infra/Storage cost + Support/Services cost
Open source packaging solution
- E.g. Kaltura’s nginx based vod packaging module (The one we discussed in detail in the previous sections)
- Build your own packaging engine using nginx and open source packaging modules
- Run it anywhere — Cloud VMs, containers, kubernetes etc.
Net Cost = Infra/Storage cost
Regardless of the kind of solution you go with, dynamic packaging will naturally lead to substantial storage savings. The more compelling reason — It will make your video delivery architecture much more flexible and future proof. Need to introduce CMAF? No need to re-encode your entire library.
In the next few posts, we will be exploring DRM, SSAI, and CDN analytics. Stay tuned and Stay Safe!