Integrated AI Infrastructure: Red Hat Inference Server & VLLM

Uniforming AI infrastructure is becoming increasingly important thanks to the willingness of artificial intelligence to increase productivity. Streamline and enhance the entire AI development lifecycle, enabling organizations to build, deploy and scale AI solutions more efficiently, safely and cost-effectively.

Red Hat AI Inference Server Meet the needs of a unified AI infrastructure as a scalable, secure and consistent platform designed to deploy, manage and deliver machine learning models across a hybrid cloud environment. The solution, according to the growing demand for a robust AI infrastructure. Brian Stevens (Photo, left), Senior Vice President and Chief AI Technology Officer, Red Hat Inc.

“The inference server is kind of core. If so, it’s equivalent to Linux, the red hat AI inference server is our chosen name, and (virtual large language model) is an open source project equivalent to the Linux kernel. It’s to stay the same so that all the innovations, accelerators, models can be delivered to users without changing. I think that what I did with VLLM (it’s) Red hat inference server will be its core platform, and everything we talked about, the right agent (model context protocol).”

With Stevens Joe Fernandez (Right), Vice President and General Manager of Red Hat’s AI Business Unit spoke to TheCube’s Rebeccanite and Rob Strechey in Red Hat SummitLiveStreaming Studio on Siliconangle Media during an exclusive broadcast on TheCube. They discussed the importance of a unified AI infrastructure and how Red Hat leads charging through AI Incerence Server and VLLM. (*Disclosure below.)

Integrate AI infrastructure through Red Hat’s VLLM project

Red Hat’s VLLM Project It plays a key role in integrating AI infrastructure by bringing scalability and enterprise readiness to large-scale language deployments. According to Stevens, VLLM can help organizations make AI advances of the future by focusing on integration with Kubernetes, support for hybrid clouds and innovation in open source.

“The way AI is heading is a very fragmented world,” he said. “Our vision is, ‘How do you unify it on a common platform like using Linux? There could be one core. VLLM that can run all models and run all accelerators?’ In doing so, think about what that means for the end user.

Red Hat takes advantage of leverage Rama Stack Developing an enterprise-ready agent AI system and integrating it to develop it OpenShift AI Platform. This integration, according to Fernandez, provides a unified framework for building, deploying and managing intelligent agents with complex inference, tool integration, and searched generation workflows.

“Meta had just released the Llama Stack as part of the launch of the Llama 3,” he said. “It was an open source license, but it was given the opportunity to work with meta and other partners who have similar interests. It becomes the core (application programming interface) of end users who want to build agents and applications on the platform. As you build new agents, you bring in new features and integrate with other capabilities, and it is integrated with other capabilities, with the agent API for tool call.”

Enabling AI models to run across a wide range of environments, including cloud, on-premises, and edge is important to facilitate deployment, adaptability and performance optimization. According to Fernandez, Red Hat supports this purpose of maximizing utility.

“Red Hat has always been a platform company,” he said. “I think AI is the next evolution, so as a platform provider, we need to enable our customers to run AI models in any model of their choice, powering their environment, accelerators and business. If you’re building new applications and not building with cloud-native and containerized architectures, it’s out of the mainstream.”

Here’s a complete video interview, part of the Silicon Angle and part of TheCube coverage Red Hat Summit:

https://www.youtube.com/watch?v=z4c0x_q-qwg

(*Disclosure: Red Hat Inc. sponsored this segment of TheCube. Neither RedHat nor any other sponsors edit content from TheCube or Siliconangle.)

Photo: Silicon Angle