RunPod Blog

RunPod Blog

Sign in Subscribe

Serverless for Artificial Intelligence and Machine Learning Workloads

Serverless for Artificial Intelligence and Machine Learning Workloads

The need to upscale, reduce operational overhead, and bring cost efficiency allows serverless computing to revolutionize AI/ML workloads. Scaling often results in expensive cost management and hardware maintenance that becomes unbearable with traditional infrastructure. RunPod dynamically allocates resources in these instances to work seamlessly with modern AI workflows. This

A Leap into the Unknown: Why I Joined RunPod

A Leap into the Unknown: Why I Joined RunPod

This entry has been contributed by Jean-Michael Desrosiers, Head of Enterprise at RunPod. I take shots—sometimes far too many, and in wildly different directions. I always have, and it’s been a part of my DNA for as long as I can remember. Picture an overly enthusiastic explorer darting

Deploy Repos Straight to RunPod with GitHub Integration

Deploy Repos Straight to RunPod with GitHub Integration

RunPod is pleased to announce its latest feature aimed at making the lives of developers easier: GitHub integration! Previously, Docker images were the primary method of deploying endpoints, and while this is still functional and useful, requires a number of intermediary steps. Now, with GitHub integration you can deploy directly

Lightricks LTXVideo: Sleeper Hit Open Source Video Generation

Lightricks LTXVideo: Sleeper Hit Open Source Video Generation

With new packages like Mochi and Hunyuan Video now out, there have been some other video packages that have come out that have also slipped under the radar that definitely deserve some more love. LTXVideo by Lightricks appears to be slept on despite coming out with an out of the

Building an OCR System Using RunPod Serverless

Building an OCR System Using RunPod Serverless

Learn how to build an Optical Character Recognition (OCR) system using RunPod Serverless and pre-trained models from Hugging Face to automate the processing of receipts and invoices. Introduction Processing receipts and invoices manually is both time-consuming and prone to errors. Optical Character Recognition (OCR) systems can automate this task by

Community Spotlight: How AnonAI Scales Its Chatbot Agents Through RunPod

Community Spotlight: How AnonAI Scales Its Chatbot Agents Through RunPod

RunPod is pleased to share the story of one of our valued clients, Autonomous. We at RunPod believe very strongly in the power of free speech and privacy - our pods are run in secure environments with optional encryption and we stand by our promise that we do not inspect

Announcing Global Networking For Cross-Data Center Communication

Announcing Global Networking For Cross-Data Center Communication

RunPod is pleased to announce its launch of our Global Networking feature, which allows for cross-data center communication between pods. When a pod with the feature is deployed, your pods can communicate with each other over a virtual internal network facilitated by RunPod. This means that you can have pods

How Much Can a GPU Cloud Save You, Really?

How Much Can a GPU Cloud Save You, Really?

Machine learning, AI, and data science workloads rely on powerful GPUs to run effectively, so organizations are deciding to either invest in on-prem GPU clusters or use cloud-based GPU solutions like RunPod. This article will show considerations of infrastructure requirements and compare the cost and performance to help you choose

Scoped API Keys Now Available on RunPod

Scoped API Keys Now Available on RunPod

We've released an expansion to our handling of API keys on RunPod. Previously, you were able to create API keys with read or read and write permissions, but now you can scope keys by endpoint and have more fine-grained control over what your keys allow access to. Here&

When to Use (Or Not Use) RunPod's Proxy

When to Use (Or Not Use) RunPod's Proxy

RunPod uses a proxy system to ensure that you have easy accessibility to your pods without needing to make any configuration changes. This proxy utilizes Cloudflare for ease of both implementation and access, which comes with several benefits and drawbacks. Let's go into a little explainer about specifically

Comparing Different Quantization Methods: Speed Versus Quality Tradeoffs

Comparing Different Quantization Methods: Speed Versus Quality Tradeoffs

Introduction Quantization is a key technique in machine learning that is used to reduce the model size and speed up inference, especially when deploying models on hardware with resource constraints. Nevertheless, achieving a good quantization setup means balancing the model performance against the computational efficiency required by the deployment environment.

Community Spotlight: How to Build and Deploy an AI Chatbot from Scratch on RunPod

Community Spotlight: How to Build and Deploy an AI Chatbot from Scratch on RunPod

In an extremely generous contribution to the RunPod community, our friends at Code in a Jiffy recently shared their journey of building a complete coffee shop application enhanced with artificial intelligence. This comprehensive project showcases how AI can transform everyday commerce applications into intelligent, interactive experiences. The video is 12

Classifier Free Guidance in LLMs - How Does It Work?

Classifier Free Guidance in LLMs - How Does It Work?

Classifier-Free Guidance (CFG) has emerged as a powerful technique for improving the quality and controllability of language model outputs. While initially developed for image generation models, CFG has found successful applications in text generation. Let's dive deep into how this technique works and why it's becoming

Mochi 1 Text-To-Video Represents New SOTA In Open Source Video Gen

Mochi 1 Text-To-Video Represents New SOTA In Open Source Video Gen

Text-to-video generation is a space where open source has lagged behind for some time, due to the difficulty and cost involved in training and evaluating video as opposed to text and images. Offerings such as Sora, while impressive, beg for open-source alternatives where you can create videos of any kind

Stability.ai Releases Stable Diffusion 3.5 - What's New in the Latest Generation?

Stability.ai Releases Stable Diffusion 3.5 - What's New in the Latest Generation?

On October 22, Stability.AI released its latest version of Stable Diffusion, SD3.5 There are currently two versions out (Large and Large Turbo), with the former geared towards quality while the latter favoring efficiency. Next week, Medium will release, aimed at smaller GPU specs. You can quickly and easily

NVidia's Llama 3.1 Nemotron 70b Instruct: Can It Handle My Unsolved LLM Problem?

NVidia's Llama 3.1 Nemotron 70b Instruct: Can It Handle My Unsolved LLM Problem?

Earlier this month, NVidia released Llama 3.1 Nemotron Instruct, a 70b model that has taken some notably high spots on various leaderboards, seeming to punch far above its weight. As of October 14th, it is not only beating high-end closed source models that far outweigh it like Claude 3

How to Code Directly With Stable Diffusion Within Python On RunPod

How to Code Directly With Stable Diffusion Within Python On RunPod

While there are many useful front ends for prompting Stable Diffusion, in some ways it can be easier to simply it directly within Jupyter Notebook, which comes pre-installed within many RunPod templates. Once you spin up a pod you get instant access to Jupyter as well, allowing you to directly

Why LLMs Can't Spell 'Strawberry' And Other Odd Use Cases

Why LLMs Can't Spell 'Strawberry' And Other Odd Use Cases

Picture this: You've got an AI language model - let's call it Bahama-3-70b - who can write sonnets, explain quantum physics, and even crack jokes. But ask it to count the r's in "strawberry," and suddenly it's like a toddler

How to Easily Work with GGUF Quantizations In KoboldCPP

Text Generation

How to Easily Work with GGUF Quantizations In KoboldCPP

Everyone wants more bang for their buck when it comes to their business expenditures, and we want to ensure you have as many options as possible. Although you could certainly load full-weight fp16 models, it turns out that you may not actually need that level of precision, and it may

Introducing Better Launcher: Spin Up New Stable Diffusion Pods Quicker Than Before

Image Generation

Introducing Better Launcher: Spin Up New Stable Diffusion Pods Quicker Than Before

Our very own Madiator2011 has done it again with the release of Better Forge, a streamlined template that lets you spin up an instance with a minimum of fuss. One fairly consistent piece of feedback brought up by RunPod users is how long it takes to start up an image

Use RunPod Serverless To Run Very Large Language Models Securely and Privately

Use RunPod Serverless To Run Very Large Language Models Securely and Privately

As discussed previously, a human interacting with a chatbot is one of the prime use cases for RunPod serverless functions. Because the vast majority of the elapsed time is on the human's end, where they are reading, procesisng, and responding, the GPU sits idle for the vast majority

Evaluate Multiple LLMs Simultaneously in a Flash with ollama

Evaluate Multiple LLMs Simultaneously in a Flash with ollama

Imagine you are a studio manager tasked with serving up a creative writing assistant to your users, and are directed to select only a few best candidates to run on endpoints to keep the project maintainable and within scope. As of the writing of this article, there are more than

Optimize Your vLLM Deployments on RunPod with GuideLLM

Optimize Your vLLM Deployments on RunPod with GuideLLM

As a RunPod user, you're already leveraging the power of GPU cloud computing for your machine learning projects. But are you getting the most out of your vLLM deployments? Enter GuideLLM, a powerful tool that can help you evaluate and optimize your Large Language Model (LLM) deployments for

RunPod Weekly #17 - Pricing Updates, SGLang Worker (Beta), Blogs

RunPod Weekly #17 - Pricing Updates, SGLang Worker (Beta), Blogs

Welcome to another round of RunPod Weekly! This week, we are excited to share the following: 📈 Pricing Updates We've been running a temporary promotion for A40 48GB GPUs, known for their exceptional combination of vRAM, performance, and pricing. We've been thrilled to see the amazing products

Run Gemma 7b with vLLM on RunPod Serverless

Run Gemma 7b with vLLM on RunPod Serverless

In this blog, you'll learn: * About RunPod's latest vLLM worker for the newest models * Why vLLM is an excellent choice for running Google’s Gemma 7B * A step-by-step guide to get Google Gemma 7B up and running on RunPod Serverless with the quick deploy vLLM worker.