Tag
This blog post describes the architecture for a scalable ingestion pipeline using Temporal to handle crawling, extracting, chunking, and embedding customer documentation from various sources, emphasizing durability, statefulness, and concurrency control.
The article analyzes the scalability limitations of using PostgreSQL as a job queue, specifically highlighting performance bottlenecks caused by MultiXact SLRU contention under high concurrency. It explains why this architecture fails in production despite working well in development and suggests considering alternatives.
An OpenAI backend engineer shares their personal journey into programming and describes their work maintaining and optimizing OpenAI's large-scale supercomputing clusters used for AI model training. The post highlights the complexity and scale of infrastructure challenges encountered at OpenAI.