@mattpocockuk: Here's the talk I gave at AI Engineer Paris: I announce /retro and /pr, and talk about how to get more PR's through you…

Here’s the talk I gave at AI Engineer Paris:

I announce /retro and /pr, and talk about how to get more PR’s through your org faster by:

• Stopping the slop
• Making PR’s easier to review (with /pr)

https://t.co/LphXCNDZCF

TL;DR: Black Forest Labs demonstrates how their video generation model, Flux 3, can be customized not only for media creation but also for controlling robots and playing video games, using a six-step pipeline focused on prompt engineering, safety configuration, and model fine-tuning.

Introduction: From Image/Video Generation to Robotics

At AI Engineer Paris 2024, a representative from Black Forest Labs presented on the versatility of their foundational model, Flux. While known for generating high-quality images and videos, the team is now adapting the same core technology to control robots and play video games.

The presentation highlights that the connection between media generation and real-world control lies in the customization of the Flux model. The pre-trained model is designed to deeply understand the visual world and the behavior of objects within it. Black Forest Labs’ AI Solutions team specializes in fine-tuning this powerful base model for specific, novel use cases.

The Flux Customization Pipeline: Six Key Steps

To achieve targeted results, Black Forest Labs employs a structured, six-step inference pipeline. This framework outlines the primary vectors for customization when working with enterprise clients.

1. User Input: The initial input from the user, which can be a text, image, or video prompt.
2. Prompt Upsampling: An LLM or Vision-Language Model (VLM) is placed between the user and the Flux model. Its role is to interpret the user’s true intent and rephrase or expand the prompt to better align with the model’s training distribution and achieve the desired output.
3. Moderation & Safety: A critical product layer that determines what content is visible or allowed. This is largely separate from the model weights but is essential for the user experience and can be customized.
4. The Model: The core Flux model weights. Customization here includes fine-tuning with LoRAs for specific styles, characters, or behaviors. It also enables adding entirely new modalities, such as action control for robotics.
5. Post-Processing: Optional steps like upscaling, restoration, or further moderation applied after initial generation.
6. Deployment & Optimization: Focuses on how the model is served, optimized, and potentially distilled for specific hardware or latency requirements.

Customization Vector 1: Prompt Upsampling

Prompt upsampling uses an intermediary LLM/VLM to bridge the gap between a user’s raw request and the model’s optimal generation point.

• Basic Example: A simple user prompt like “cabin in the forest” might generate a generic image. An upsampler can expand it to “hyper-realistic, right angle, stone chimney, soft golden glow,” guiding the model toward a more specific aesthetic based on learned user preferences.
• Enterprise Case Study: A client needed to generate illustrations for random, machine-generated conversation topics (e.g., “family-friendly travel destinations” or “draining a dishwasher”) within their app. The key challenge was adhering to strict design requirements: the subject must be on the right, the image must be clean, and ample negative space on the left for text overlay.
  • Without upsampling: Outputs were nonsensical and varied wildly.
  • With a generic upsampler: Outputs leaned toward illustration but failed design rules.
  • With a custom-configured upsampler: The system prompt was engineered to understand both the user’s intent and the firm design rules, consistently generating images that placed the subject correctly and maintained clean composition.

This method is powerful because it is highly iterative (requires only prompt engineering, no retraining) and offers maximum flexibility. It directly addresses user laziness by translating vague requests into precise model instructions.

Customization Vector 2: Moderation and Safety Configuration

Moderation is a fundamental product layer that defines the user experience by allowing or blocking specific content. Black Forest Labs offers two primary methods for customization:

1. Configuration-Based (API Settings): A modular system allows granular control via parameters:
   • Safety Tolerance: A general setting for content strictness.
   • Category Control: Allow or block specific categories like violence, sexual content, self-harm, intellectual property (IP), and likenesses of real people.
   • Block/Allow Lists: Custom lists for specific use cases (e.g., a children’s platform blocks all sensitive categories; a media studio might allow specific licensed characters while blocking all others).
2. Embedded in Model Weights: For deployments on client infrastructure where the full moderation stack is not provided, the safety and moderation rules are directly embedded into the model’s weights themselves.

This customization is crucial for diverse clients. For instance, a fashion retailer might allow more suggestive content but block competitors’ brands, while a film studio might require flexibility for licensed violent content but strictly enforce IP for unlicensed characters.

Source: https://www.youtube.com/watch?v=g0vqT_wZtXA