@robertnishihara: I watched a bunch of the talks at Ray Summit live and this one by Priunsh Syen and Tyler Titsworth at @LilaSciences abo…

I watched a bunch of the talks at Ray Summit live and this one by Priunsh Syen and Tyler Titsworth at @LilaSciences about the details of their AI research platform was extremely impressive.

https://t.co/VVfZpUQUbp

TL;DR: Lila Sciences built an internal self-service AI research platform to support their “build-test-learn” loop for scientific discovery, using a combination of Flight, Ray, and GitOps to handle diverse workloads while reducing friction and maintaining security for researchers.

Introduction: The Goal of a Scientific Super-Intelligence Platform

Lila Sciences is building automated laboratories to handle large-scale experiments. Their core philosophy is the “build-test-learn” loop, which they aim to optimize with Large Language Models (LLMs). They describe this as an inner loop (design-build-test-learn) and an outer loop (training models to optimize the process itself), believing this path will lead to a scientific super-intelligence to solve humanity’s greatest challenges.

This requires massive deep learning resources, primarily GPU compute, and a platform to manage it. The central question for their platform team is: How do you build a self-service research platform? The platform must support both loops, empower scientists to use AI models, and allow them to contribute back to the platform itself.

The ultimate objective is to reduce compute execution time and optimize ideation time, enabling researchers to focus on science. This must be done in a cloud-native environment with containerization and other operational patterns, allowing scientists who aren’t production code experts to scale their capabilities.

From Initial State to Future Vision

At inception, the platform focused on specific initial directions. However, the long-term vision is far broader and requires supporting:

• Multi-cluster, multi-cloud, and multi-region deployments.
• Beyond just LLM supervised fine-tuning (SFT), including Reinforcement Learning (RL).
• Expanding beyond protein generation to physical simulations, training domain-specific models, and developing evaluation suites and benchmarks.
• Managing cross-region GPU clusters.
• Standardizing all these workflows to reduce the maintenance burden of submitting and running tasks.

Platform Architecture and Core Principles

The initial architecture addressed a fundamental problem: different teams were using different AWS services, vendors (like Weights & Biases), and Kubernetes-based neoclouds. The platform needed to integrate these and provide a unified execution layer over all of them.

The platform’s middle layer—the GPU Plane and Control Plane—is built on non-negotiable principles:

• Fair GPU Scheduling, Observability, and Orchestration: Essential for allowing multiple users to train models concurrently.
• Multi-tenancy and Policy Enforcement: Necessary for security teams to isolate model serving and enforce policies. These principles define the platform’s needs before any specific software decisions are made.

Verticals and Software Decisions

The team divided the platform into three core verticals based on the nature of the workloads:

1. Workload Execution: For anything that is task-like.
2. Model Serving: For non-deterministic services, typically model inference.
3. Agents: Requiring a safe, sandboxed environment.

The Software Stack: Flight, Ray, and GitOps

• Flight (v1): Provides the foundation. It combines with Ray for DAG (Directed Acyclic Graph) orchestration across multi-cluster, multi-cloud environments. Ray acts as a scalable sub-orchestrator for training workloads, batch inference, and ETL pipelines.
• Model Serving: Uses a GitOps pipeline with Argo CD (with limited permissions) and Ray LLM Serve, which is well-suited for sandboxed environments.
• Agent Sandboxing: Utilizes Jupyter kernels, Python kernels, and a basic shell.
• Foundation Components: These verticals are supported by OPA (Open Policy Agent) and Kyverno for policy execution, combined with identity providers to manage organization-level permissions.

The core decision was to define clear “golden paths” for each vertical. For example, model serving is now defined via a YAML file submitted through a PR to GitHub, while workload execution has a unified submission API.

Reducing Friction with Flight

Flight is key to reducing initial friction for researchers:

• Automatic Containerization: Flight automatically generates a Dockerfile, build/push commands, Kubernetes manifests, and handles deployment, all through a Pythonic interface. A researcher only needs to specify dependencies and a base image.
• Single Command Execution: The process can be started with a single command: pyflight run.
• Cloud-Based Container Builds: To avoid slow local uploads, all task builds are forwarded to powerful, remote builders within the EKS cluster using the Moby project’s BuildX plugin.
• Declarative Assets: For model serving, the platform uses declarative tools like Argo CD and Crossplane, allowing entire infrastructure and deployment pipelines to be defined in YAML. This includes deploying the Ray Service and all necessary components like ingress configuration to make the model accessible from a notebook.

Getting Researchers on Board: The “Golden Path”

Once deployed, the challenge is getting researchers to use the platform effectively. The process is abstracted into four stages: Ideation, Development, Security Boundary, and Self-Service (Computation). A critical focus is the security boundary, ensuring that compute entering the cluster is safe for both Lila and its clients.

The platform team’s goal is to make the standard “green path” so frictionless and beneficial (providing security, GPU quotas, observability, reproducibility) that it becomes the obvious choice, discouraging “purple” (custom) or “red” (offline) paths.

Problem and Solution: Scaling Map Tasks

An early challenge involved “Map” tasks with tens of thousands of small, GPU-dependent work items.

• Initial Issue: Using Flight directly, each item became a Pod. Scheduling 10,000 Pods created a self-inflicted DDoS on the cluster’s control plane and led to scheduling failures, as the scheduler couldn’t find correctly-sized resource pockets.
• Solution: The workflow was redesigned to use Ray. The fan-out happens inside the Ray cluster. The Kubernetes controller only sees a few Ray nodes, not thousands of Pods. The work is modeled as Ray Actors within a pre-warmed Ray Data-like abstraction, allowing efficient distributed processing within the cluster. This pattern also solves similar scaling issues for SFT and RL jobs.

Problem and Solution: Simplifying Access and Debugging

Researchers faced significant friction in simply running a task:

• Complex Authentication: A long checklist of credentials (AWS profiles, VPN, Kubernetes context, Flight tokens) where missing one step broke the entire chain.
• Opaque Pipeline: With many architectural layers (AWS, K8s, ECR, Flight), diagnosing a failure was difficult. The error might originate in one layer but manifest in another.
• Lack of Unified State: It was hard to tell if a task was queued, running, or failed, especially for resource-intensive jobs waiting for GPUs. While data existed in Grafana/Prometheus, it wasn’t consolidated for the researcher.

The Solution: Chariot

To solve this, the team built Chariot, an internal command-line tool that serves as a one-stop shop for all platform tools, simplifying authentication, task submission, and status checking.

Conclusion

Lila Sciences built their AI research platform by first defining clear core principles (fair scheduling, security, observability), then selecting and integrating tools (Flight, Ray, Argo CD, OPA/Kyverno) into logical verticals. They focused relentlessly on reducing researcher friction through automated containerization, cloud builds, and a simplified CLI (Chariot), while solving critical scaling issues by leveraging Ray for distributed workloads. The platform is a work in progress, continually evolving to support the company’s ambitious goal of a scientific super-intelligence.

Source: https://www.youtube.com/watch?v=eHb9Z6AxdMk&list=PLZNaJyYZRPdo&index=4