Three years. That’s the time that it took the IT team at French TV provider Molotov to put in place the infrastructure needed to broadcast 170 channels in 30 formats to several hundred thousand users simultaneously.
Fundamental to the platform is Scality object storage and Dell EMC server hardware.
“The big challenge was to achieve real-time encoding of 30 standard formats for every channel, each one suited to different bandwidths to suit the different device types and internet connections of our users,” says Alexandre Ouicher, technical director at Molotov.
“We set ourselves the requirement to tolerate three minutes between a channel streaming and us being able to distribute it. The key challenge is to allow our users to move from one channel to another in less than 200 milliseconds,” adds Ouicher, who is head of a team of 40, around half of the total at the company.
Molotov’s aim has been to reinvent television since 2016, and it is available via mobile, Mac, PC and set-top box.
Its mission has been to put an end to the need to zap between channels on the remote. Instead, the Molotov interface provides a range of selection options that allow the customer to choose their viewing by channel, genre, current trends, available catch-ups, upcoming attractions and to select reminders for forthcoming programmes.
Molotov’s base offer is free of charge. Paying subscribers can also switch between devices and record programmes. Molotov retains those recordings without time limit and without users having to clean up the list. Like other broadcasters, Molotov sells access to themed packages such as for films or sport.
244 servers in 11 racks
With half of its staff working on the technical side, Molotov is something of a digital powerhouse. Its datacentre comprises 244 servers in 11 racks dedicated to 24/7 transformation and distribution of programming in numerous formats.
Developers constantly refine their algorithms and have even automated procedures to maintain activity in case of an incident. Engineers watch for the slightest drop-off in performance, and with all the applications built as microservices across 240 virtual machines, load balancing takes place to account for relative demand for programming.
Video streams are relayed as close as possible to users by content delivery networks (CDNs), to cache servers that dedicated providers install in access providers’ datacentres.
These servers have to absorb customer demand during programme delivery, and also need to be able to provide access to the whole range of programming available.
“We retain all the programming we deliver so that viewers can restart them from the beginning, but also because we offer the ability to resume viewing later on. For that reason we need storage with a lot of capacity, which is scalable and with quick read times.”
Currently, Molotov’s stored data represents around 6PB and grows by 25% to 30% per annum.
“Fundamentally, we needed to put in place a way for hundreds of thousands of users to connect simultaneously,” says Ouicher.
This part of the service is dependent on the database that feeds the user interface that customers control from their terminals.
Plug-and-play servers from Dell EMC
In 2013, to help design the infrastructure that would allow it to underpin its service three years later, Molotov called on a specialist in the area, namely the Paris-based integrator Iguane Solutions.
Iguane was founded in 2000 by Benjamin Bejbaum, who went on to create the video-hosting site Dailymotion. Also among its achievements is Deezer, the French online music rival to Spotify.
The provider advised Molotov to base its infrastructure estate on Dell EMC hardware.
“Our idea was to limit the number of products, to simplify administration and to support growth in activity by the quick and simple addition of new nodes,” says Iguane chief operating officer (COO) Cédric Le Moan.
“Iguane had chosen Dell EMC because its servers are plug-and-play. There’s not even a need to use a screwdriver. It all just slides in on rails.
“Elsewhere, Dell EMC machines are equipped with iDRac controllers, which put all server information on a central console and from where you can take control of any node,” adds Le Moan.
The COO says the challenge for an important infrastructure such as that at Molotov is to not fall into the time-consuming trap of administration.
2U rack servers dedicated to stream encoding
For stream encoding, Molotov invested in Dell EMC PowerEdge R700s. Their advantage is that they fit into 2U of rackspace with enough room to add GPU cards that are needed to provide the processing power for video format conversion.
“The initial option was to use dedicated video transcoders, but that wasn’t a satisfactory solution,” says Ouicher. “The GPU cards gave us more flexibility in application development. Iguane helped us make models to evaluate the number of GPUs and in how many servers our applications needed to process what quantity of streams.”
Ouicher doesn’t want to disclose the details of the R700 estate deployed to date, saying that it constitutes a business secret.
User connections are handled by Dell EMC R400 servers. They’re largely the same as the R700s, but fit into 1U of rackspace and can’t have a GPU added.
The R400s execute applications connected to databases and it’s the amount of RAM that governs the number of virutal machines (VMs) and the connecting network that are important.
Object storage from Scality for minimal latency
The part of the infrastructure that Alexandre Ouicher pays most attention to is storage.
Molotov decided on Dell EMC disk shelves (JBODs) that hold 12 hard disk drives (HDDs), each of 14TB in capacity, connected to PowerEdge R640 server hardware. These twin-processor 1U nodes have two Fibre Channel cards and each is capable of controlling eight JBODs.
“This part of the infrastructure is particularly scalable. The advantage of the R640 is that it is simple to add RAM to, or even to change processors, so that we always have enough power to manage capacity that always grows,” says Ouicher.
“We add to the storage and other resources three times a year, knowing that we will never go beyond eight JBODs per server due to questions of security.”
On the software side is object storage, managed from the R640 servers by Scality Ring.
Scality is particularly adapted to Molotov’s workloads. It is a system that suffers least from latency where writes are more numerous than reads.
Effectively, Molotov has to have all streams from all channels ready to transmit, in all formats, even though subscribers don’t consume them in a systematic way. Reads are uniquely those streams that must be sent via the CDN.
“I’d also add that Scality allows us to lose up to 30% of JBOD capacity without losing data,” says Ouicher. “To optimise our storage we don’t need data deduplication. Instead, we just have to regularly rebuild Scality’s index.”
Again, Ouicher refuses to be drawn on details, but he explains that to limit any explosion of data that results from users not cleaning up queued programmes on their watch lists, the Molotov IT team reorganises things, even to the point of re-encoding the oldest files.
Molotov’s infrastructure is monitored in several ways. Youbora provides specialised video stream monitoring that’s capable of showing connections between datacentre activity and user experience.
Meanwhile, applications that are built as micro-services are monitored by Datadog. Then, all business and technical metrics, system logs included, are covered by ELK’s elastic search and analytics.
In the case of other technical snags, the technical team writes Python scripts to help avoid any issues. Molotov points to monitoring the expiry of SSL certificates as an example of this.
Thanks to such monitoring capability, the “natural” security of object storage and the quality of the infrastructure, Molotov hasn’t experienced an outage since 2016.
“The only notable problems have been on the internet side of the network,” says Ouicher. “But even that wasn’t catastrophic because everything went via the CDNs.”
According to Ouicher, Molotov’s infrastructure isn’t difficult to administer. Iguane Solutions carries out some of these tasks, such as regular additions to hardware resources at its datacentre.
Molotov, meanwhile, occupies itself with system administration operations. Among all this, the backup load is lightened with regard to the volume of data in production. And for good reason: only the databases connected to user activity and the programme catalogue are protected. The rest of the data stored – the videos – are already a backup copy.