Gemini Distillation Service

# Gemini Distillation Service Source: https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/tuning/distillation?hl=he **Gemini Distillation Service** (distillation) allows users to train a smaller, more efficient 'student' model that uses the outputs and reasoning patterns of a larger, higher-capability 'teacher' model. Frontier models define the cutting edge of AI, but can be over-provisioned for specific enterprise use cases. Distillation bridges this gap, enabling production-level efficiency (lower latency and lower cost), while allowing smaller models to achieve deeper reasoning capabilities. Unlike standard supervised fine-tuning (SFT), which only uses the final text output, distillation uses: - **Teacher answers**: The final text output. - **Raw thoughts**: The internal reasoning paths generated by the teacher model. ## Supported Models The following models are supported for distillation during the early access phase: - **Teacher model**: `gemini-3.1-pro` - **Student model**: `gemini-2.5-flash` Flow chart illustrating a 5-step AI model distillation process. Step 1: Input Dataset (GCS) receives a prompt dataset (JSONL) and an optional validation dataset. Step 2: Data Ingestion and Preparation includes format validation and sharding for parallel processing. Step 3: Teacher Model Inference uses the teacher model (Gemini 3.1 Pro) to generate a high-quality labeled dataset, consisting of detailed answers and internal reasoning paths. Step 4: Distillation Training uses the student model (Gemini 2.5 Flash), which learns to imitate the teacher's thought process using answers, reasoning patterns, and distributions. Step 5: Output: Distilled Model Generation produces three items: a final distilled model registered in the Model Registry, 10 intermediate checkpoints registered for evaluation, and Agent Platform endpoints automatically generated for the checkpoints. Figure 1. System illustration showing how the Gemini Distillation Service works. ## Suitable Use Cases Distillation is recommended over standard prompting or supervised fine-tuning (SFT) in the following scenarios: - **High-volume, latency-sensitive applications**: If your application requires the reasoning capabilities of a Pro-level model, but must meet strict latency SLAs or budget constraints that necessitate using a Flash-level model. - **No ground truth data (SFT not possible)**: If you have a large dataset of prompts or user queries, but lack the resources to manually label or generate high-quality ground truth answers required for standard SFT. - **Complex reasoning tasks**: Tasks involving multi-step logic, summarizing highly technical help articles, or complex programming tasks, where the base Flash model struggles but the Pro model succeeds. - **Significant performance gap**: When the teacher model substantially outperforms the base student model on your specific task, providing a clear knowledge gap to transfer during distillation. ## Prerequisites and Project Setup Before starting a distillation job, ensure your Google Cloud environment is correctly configured: 1. **Request access to the allowlist**: Ensure your Google Cloud project ID is added to the allowlist for early access to the Gemini Distillation Service. To add your project to the allowlist, contact your Google sales representative. 2. **Enable the API**: Enable the Agent Platform API in your Google Cloud project. 3. **Configure IAM role permissions**: You must have the Agent Platform Admin role (`roles/aiplatform.admin`) in IAM. 4. **Set the region**: Distillation jobs must be run in the `us-central1` region. ## Dataset Preparation A key feature of this service is the use of **prompt-only datasets**. Since the teacher model generates the target output during the distillation process, you do not need to provide expected answers. ### Dataset Requirements Datasets must be in JSON Lines (JSONL) format and stored in a Cloud Storage bucket. Each record must follow the format of a Gemini custom tuning dataset (https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/gemini-supervised-tuning-prepare?hl=he#dataset-example), in addition to the following format: - **System instructions**: An optional `systemInstruction` field (with role 'system') can be included to define system prompts. - **Input**: The content field (with role 'user') is required for the primary input. - **Multi-turn prompts**: Roles 'user' and 'model' can be alternated, provided the last value in the sequence is 'user'. Two examples of `dataset.jsonl` files: `` { "contents": [ { "role": "user", "parts": [ { "text": "You're the artist here. Choose as many strands of thread as you like, as long as you're using three or more. Go for color combinations that you think would make a pretty pattern. Get creative! If you only use one color of thread, you won't be able to create a pattern.\n\nProvide a summary of the article in two or three sentences:\n\n" } ] } ] }, `` `` { "contents": [ { "role": "user", "parts": [ { "text": "You're the artist here. Choose as many strands of thread as you like, as long as you're using three or more. Go for color combinations that you think would make a pretty pattern. Get creative! If you only use one color of thread, you won't be able to create a pattern.\n\nProvide a summary of the article in two or three sentences:\n\n" } ] }, { "role": "model", "parts": [ { "text": "Choose several strands of embroidery thread in a variety of colors." } ] }, { "role": "user", "parts": [ { "text": "You will need one egg (raw or hard boiled but hard boiled is best) and one spoon for each person participating in the race. You might even like to use dyed Easter eggs as something special for Easter. It's best to have this race on grass or some other soft surface, to give dropped eggs a chance!" } ] } ] } `` ### Best Practices When creating your dataset, follow these guidelines: - **Size**: Provide at least 1,000 examples to see a noticeable quality improvement. - **Diversity**: Ensure the prompts include edge cases and varying lengths expected in your traffic. ## Configuring the Distillation Request To run a distillation job, configure the generation behavior of the teacher model and the training hyperparameters of the student model. ### Configuring the Teacher Model's Behavior You need to configure how the teacher model responds to the dataset. The quality of the student model directly depends on the quality of the teacher model's output. To configure the teacher model's generation behavior, set `candidateCount`: - `candidateCount`: Number of response variations to generate. (Example: `4`. Range `[1, 5]`). If not specified in the request, the system uses the default value `4`. ### Configuring Distillation Hyperparameters The distillation hyperparameters control the training process of the student model. More information on hyperparameters in Gemini Enterprise Agent Platform is available in the section Creating a Tuning Job (https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/gemini-use-supervised-tuning?hl=he#create_a_text_model_supervised_tuning_job) in the Supervised Fine-Tuning (SFT) guide. When creating a distillation job, configure the following hyperparameters: - `epochCount`: Number of times the student model iterates over the dataset. (Example: `20`. Range `[1, 100]`). If not specified, the system uses the default value `4`. - `learningRateMultiplier`: Adjustment to the base learning rate of the student model. (Example: `2.0`. Range `[0.25, 4]`). If not specified, the system uses the default value `1`. ## Starting the Distillation Job During the early access period, you can submit and monitor distillation jobs using the Agent Platform API REST version. You can initiate a new distillation job (https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/tuning/distillation?hl=he#create-a-new-distillation-job) or perform continuous tuning (https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/tuning/distillation?hl=he#perform-continuous-tuning) on a checkpoint of an already distilled model. ### Creating a New Distillation Job Create a JSON file named `request.json` containing the job configuration. In the following example, the teacher generation configuration is embedded in the `hyperParameters` field: `` { "description": "Distillation testing job.", "baseModel": "gemini-2.5-flash", "tunedModelDisplayName": "flash-distillation-run-1", "distillationSpec": { "promptDatasetUri": "gs://your-bucket/path/to/prompt_dataset.jsonl", "validationDatasetUri": "", "base_teacher_model": "gemini-3.1-pro-preview", "hyperParameters": { "epochCount": "20", "learningRateMultiplier": 2.0, "generation_config": { "candidateCount": 5 } } } } `` Submit the job using `curl`: `` curl -X POST \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $(gcloud auth print-access-token)" \ https://us-central1-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/us-central1/tuningJobs \ -d @request.json `` ### Performing Continuous Tuning If you want to continue the tuning process from a checkpoint of a previously distilled model, include the `preTunedModel` block in the `request.json` file. Continuous tuning is only supported for checkpoints of previously distilled models, with the same student base model. Checkpoints of models that underwent supervised fine-tuning (even with the same student base model) are not supported. The following example shows how to configure continuous tuning for a distilled model checkpoint: `` { "description": "Continuous distillation testing job.", "preTunedModel": { "tunedModelName": "projects/YOUR_PROJECT_ID/locations/us-central1/models/PRETUNED_MODEL_ID@1", "checkpointId": "1", "baseModel": "gemini-2.5-flash" }, "tunedModelDisplayName": "flash-distillation-continuous", "distillationSpec": { "promptDatasetUri": "gs://your-bucket/path/to/prompt_dataset.jsonl", "validationDatasetUri": "", "base_teacher_model": "gemini-3.1-pro-preview", "hyperParameters": { "epochCount": "20", "learningRateMultiplier": 2.0, "generation_config": { "candidateCount": 5, } } } } `` Submit the payload using `curl`: `` curl -X POST \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $(gcloud auth print-access-token)" \ https://us-central1-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/us-central1/tuningJobs \ -d @request.json `` ## Monitoring the Distillation Job The response to the request will return a job name containing `JOB_ID`. You can check the job status (`state`, errors, and final hyperparameters) by sending a GET request: `` curl -X GET \ -H "Authorization: Bearer $(gcloud auth print-access-token)" \ -H "Content-Type: application/json; charset=utf-8" \ https://us-central1-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/us-central1/tuningJobs/JOB_ID `` You can also monitor progress visually in the Google Cloud console. To do this, navigate to **Agent Platform > Tuning** and select the `us-central1` region. In this early access version, the Agent Platform Console UI has the following known limitations: - **Teacher model sampling progress**: No progress widget for the teacher model sampling process. The status may show "Preparing to run tuning", but the job will execute in the background as normal. - **Student model tuning charts**: During the student model tuning phase, the UI provides loss curve charts and total training text tokens. - **Checkpoint table**: The UI displays a table of intermediate checkpoints and links to the prediction endpoint created in Agent Platform for evaluation. The Epoch column in this table displays the value 0 due to a known issue. ## Cancelling the Distillation Job To cancel a currently running distillation job, perform one of the following actions: - Use the console and modify the following URL: `https://console.cloud.google.com/agent-platform/tuning/managed?project=YOUR_PROJECT_ID&vertex_ai_region=us-central1`. Replace `YOUR_PROJECT_ID` with your project ID. - Use `curl` to send a POST request to cancel the job: `` curl -X POST \ -H "Authorization: Bearer $(gcloud auth print-access-token)" \ -H "Content-Type: application/json; charset=utf-8" \ https://us-central1-aiplatform.googleapis.com/v1beta1/projects/YOUR_PROJECT_ID/locations/us-central1/tuningJobs/YOUR_JOB_ID:cancel `` Replace the following fields: - `YOUR_PROJECT_ID` with your project ID. - `YOUR_JOB_ID` with the job ID. ## Evaluating the Result After the distillation job completes successfully, the new student model is automatically registered in the Gemini Enterprise Agent Platform model registry, and one or more dedicated endpoints are created to serve predictions in production. To evaluate the result, locate the endpoint, send a prediction request, and perform evaluation. To evaluate the result: 1. Send the following GET request to see the status of the tuning job: `` curl -X GET \ -H "Authorization: Bearer $(gcloud auth print-access-token)" \ -H "Content-Type: application/json; charset=utf-8" \ https://us-central1-aiplatform.googleapis.com/v1beta1/projects/YOUR_PROJECT_ID/locations/us-central1/tuningJobs/YOUR_JOB_ID `` Replace the following fields: - `YOUR_PROJECT_ID` with your project ID. - `YOUR_JOB_ID` with the job ID. 2. Completed jobs show a nested `endpoint` field inside the `tunedModel` object. Look for `ENDPOINT_ID` at the end of the returned path string (e.g., `projects/.../endpoints/YOUR_ENDPOINT_ID`). Record the endpoint ID. Ensure the tuning job completed successfully, as the endpoint is not available while the tuning job is still running or has failed. If the `endpoint` field is missing, debug the tuning job by examining its `state` or `error` keys. 3. Create a JSON payload file named `generate_content_request.json` containing the prompt: `` { "contents": { "role": "user", "parts": [ { "text": "hi, say something" } ] } } `` 4. The following POST request example shows how to send a prediction request: `` curl -X POST \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $(gcloud auth print-access-token)" \ https://us-central1-aiplatform.googleapis.com/v1beta1/projects/YOUR_PROJECT_ID/locations/us-central1/endpoints/YOUR_ENDPOINT_ID:generateContent \ -d @generate_content_request.json `` Replace the following fields: - `YOUR_PROJECT_ID`: your project ID. - `YOUR_JOB_ID`: the job ID. - `ENDPOINT_ID`: the endpoint ID. 5. To evaluate the results: 1. Run a holdout test dataset, using prompts not included in the training data, against the new distilled model. 2. Compare the output to the base `gemini-2.5-flash` model to measure quality improvements. 3. Compare the output to the `gemini-3.1-pro` model to determine how close the student is to the teacher's reasoning. ## Limitations The following table describes the limitations of distillation: Distillation is subject to the following limitations: - **Model limitations**: - List of supported models (https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/tuning/distillation?hl=he#supported-models) - **Dataset limitations**: - **Volume limitations**: - Maximum capacity for a training dataset is 50,000 examples. - The source JSONL file size must not exceed 1GB. - **Context window specification**: - The service can accept up to 8,000 input tokens per record. If more than 10% of the provided records exceed this threshold, distillation jobs are stopped. - Teacher model sampling is limited to output of up to 24,000 tokens. In cases where the teacher model generates more than 24,000 tokens, content is truncated at this limit, which may affect student model performance. - **Modality**: Limited to text-based data. Multimodal input, including video, images, or function calling requests, is not supported. - **Configuration and hyperparameter limitations**: - When configuring `distillationSpec` and its associated parameters, adhere to the following restrictions: - Encryption: CMEK cannot be used for distillation jobs involving models from direct Google interaction. - `epochCount`: The value must be an integer between 1 and 100. - `learningRateMultiplier`: Values must be in the floating-point range `0.25` - `4.0`. - **Single-stage distillation**: Teacher model sampling and student model tuning are performed in a single API call. If you have a large amount of data to sample, the same data must be sampled again during subsequent tuning. ## Access If you would like to try the Gemini Distillation Service, you can contact the Tuning Service team at [[email protected]](mailto:[email protected]) to request access and be added to the project allowlist. To ensure optimal performance and efficient resource management, it is recommended to create a dedicated Google Cloud project for distillation jobs. When contacting our team, it is helpful to specify your project ID or project number to expedite the allowlist process. ### Quota and Policy