3 Reasons why LOVO AI Switched to RunPod Serverless

Since April 2023, LOVO AI 10x’d their usage on RunPod Serverless after switching from another leading cloud provider.

Why the rapid increase? According to Hara Kang, CTO at LOVO:

“RunPod is unparalleled in 3 key areas for machine learning inference: network storage, flexible autoscaling and developer experience”

About LOVO

LOVO AI is a Berkeley founded AI voice generation platform. Their voice cloning and text-to-speech products are hyper-realistic and diverse. They create tools that make it dead simple for media and marketing teams to create product videos, slashing costs for small teams that would have otherwise had to hire voice actors and script writers.

The team at LOVO has an incredibly strong expertise in building foundational voice models. When they were ready to launch after years of research, they were mainly looking for an inference solution with simple setup and didn’t want to go through the hurdles of setting up their own infrastructure.

Instead of hiring a dedicated ML ops team, they decided to try out RunPod Serverless to support the complex custom workload needed to deploy their foundational AI voice model.

1) RunPod's easy-to-use network storage

The LOVO team’s use case was unique since they needed to support a large number of custom voices for over 100 different languages. To achieve this, they use 5 TB of network storage across 30+ volumes. According to LOVO’s CTO, Hara Kang, “Runpod stands out most when it comes to network storage, since many GPU providers have network storage that’s hard to use at our scale. When you have to hold dozens or more checkpoints like we do in order to serve over 500 different voices, it can get pretty complicated.” 

With up to 100TB NVMe SSD network volumes across 4 regions and >100Gbps data transfer speeds, even storing hundreds or thousands of different models in volumes and loading them into cache is fast and straightforward on Serverless.

“There’s ways to circumvent other providers storage limitations but attaching a network volume to a pod, preparing the volume, and attaching this to other Serverless endpoints has been quite an intuitive process for us.”

— Hara Kang, CTO

2) Fast and flexible autoscaling

With over 2 million users, reliability is a necessity for LOVO. Having workers distributed over multiple regions is vital for ensuring high reliability. Over the last year, they’ve consistently been able to scale up from 10s of GPU workers to peaks of over 200 concurrent workers with Serverless. To handle their large request volume while keeping costs low, they’ve leveraged Serverless’ support of custom idle timeouts and queue delay for autoscaling. 

• With idle timeout, If a GPU worker remains idle (no incoming requests) for a specified period of time (the idle timeout), it is automatically scaled down and terminated.
• With queue delay, LOVO can set how long a request should wait in-queue before a new flex worker is provisioned to process it. 

This autoscaling strategy allows for low latency by provisioning flex workers when needed while making sure that the lower cost 24/7 active workers process most of the incoming traffic when possible.

3) Great developer experience

When LOVO started using RunPod, they noticed it was almost uncanny how often they’d realize they needed a feature, and a few days after it would be available on the platform.

“It really shows that RunPod is made by developers. They know exactly what engineers really want and they ship those features in order of importance.”

—  Hara Kang, CTO at LOVO

LOVO cited “deployment settings, interactive shell, team management, monitoring tools, execution timeout settings” as being particularly stellar compared to other AI cloud platforms.

“There are definitely providers who offer much cheaper pricing than Runpod. But they always have an inferior developer experience. If you’re paying 50% less for a GPU elsewhere, that cost is coming out somewhere else, be it developer time or lack of reliability. For the value, Runpod provides competitive prices and we’re willing to pay a premium to reduce the headache that normally comes with ML ops.”

—  Hara Kang, CTO at LOVO

RunPod looks forward to supporting LOVO as they look to scale their Machine Learning inference. If you’re in the same boat as them, try out RunPod Serverless here or book a call to learn more.