Tag
GitHub Models has been retired, causing disruptions for developers using its unified LLM API in GitHub Actions. Simon Willison reflects on the shutdown and his migration to OpenAI's API.
Analyzes how per-token LLM pricing creates incentives for verbose output, and proposes low-entropy prompt constraints (FAOA) to reduce cost and increase semantic density.
A detailed measurement study across Anthropic, OpenAI, Gemini, and DeepSeek finds that the conventional 30-second prompt cache keepalive is 8x too frequent; a 4-minute interval is optimal, and only Anthropic's cache saves money at long idle gaps.
Proposes Cache-Aware Prompt Compression (CAPC), a method that combines query-agnostic compression with caching to reduce LLM API costs, demonstrating significant savings over existing approaches on Anthropic's Sonnet API and production workloads.
Sakana AI launched Fugu, a multi-model orchestration API, after Anthropic suspended access to Claude Fable 5 and Mythos 5. Fugu Ultra scored 93.2 on LiveCodeBench, outperforming Fable 5, and starts at $5 per million input tokens.
Bash4LLM+ is a lightweight, dependency-free Bash wrapper for LLM APIs, offering secure and auditable interaction with Groq and other providers, with features like dynamic model lists, streaming, and extensible extras.
FreeLLMAPI is an open-source tool that aggregates the free quotas of 16 LLM providers into a single OpenAI-compatible endpoint, with automatic routing and usage tracking, totaling about 1.7 billion tokens per month.
The author notes a surprising lack of publicly available APIs for medical-oriented LLMs, despite models like MedGemma and BioMistral existing on Hugging Face, and seeks information on any available options.
The author explores encrypted reasoning blobs in LLM APIs from OpenAI and Anthropic, discussing how chain-of-thought data is encrypted and signed, and the security implications of tampering with those blocks.
A practical guide listing 10 strategies to reduce costs when using LLM APIs, including model selection, prompt caching, batch processing, and monitoring expenses.
FreeLLMAPI is an open-source tool that aggregates free tiers from 11 major LLM providers into a single OpenAI-compatible endpoint, routing requests and managing rate limits to deliver ~1B+ tokens per month. It simplifies access to multiple free models through one local server.
OpenAI introduces Structured Outputs in their API, enabling developers to reliably get valid JSON schema outputs from language models, improving integration with downstream systems and reducing parsing errors.
Pi is an open-source mono repo containing a coding agent CLI, agent runtime, and unified multi-provider LLM API, designed for building interactive coding agents.