Streaming data infrastructure for modern workloads [Video]

In today’s cloud-native era, efficient streaming data infrastructure is pivotal for scaling artificial intelligence training and operational insights.

As organizations increasingly rely on streaming data for artificial intelligence training, analytics and operational insights, the challenges of scaling technologies such as Apache Kafka are coming into sharp focus. These include escalating costs, operational complexities and the inefficiencies of legacy architectures in cloud-native environments, according to Akshay Shah (pictured), chief technology officer of Buf Technologies Inc.

“Here in the cloud-native community, we expect Kubernetes to be our abstraction for compute and then, for the most part, we expect object storage to be our abstraction for disks,” Shah said. “If there’s going to be cross-region replication, or if I want really fast transfer and I want RDMA instead of kernel networking, I want that handled in MinIO or Google Cloud Storage or S3. I don’t want to be in the poor Apache Kafka code base in Java trying to eke out small performance wins in the replication layer.”

Shah spoke with …