How Compiler Explorer Runs on AWS in 2026

# How Compiler Explorer Runs on AWS in 2026 — Matt Godbolt’s blog Source: [https://xania.org/202608/how-compiler-explorer-runs-on-aws](https://xania.org/202608/how-compiler-explorer-runs-on-aws) Written with LLM assistance\. Details at end\. I’ve been meaning to write an update on how[Compiler Explorer](https://godbolt.org/)actually runs on Amazon’s cloud, and it’s been sat on my list for a good while now, somewhere behind the other random things that take up what laughably I refer to as my spare time[1](https://xania.org/202608/how-compiler-explorer-runs-on-aws#fn:list)\. The last time I wrote about this was[2016](https://xania.org/201609/how-compiler-explorer-runs-on-amazon), when the whole site was a load balancer, a couple of instances and some Docker containers I built on my laptop\. I wrote a much longer[how it works](https://xania.org/202506/how-compiler-explorer-works)last summer, but that one is mostly about Compiler Explorer and only incidentally about the cloud it sits on\. It’s no secret that we run on AWS, and none of it is hidden: the[infra repository](https://github.com/compiler-explorer/infra)has all the terraform, the install scripts and the`ce`command line tool we drive the whole thing with, so if you’d rather read the real thing than my description of it, help yourself\. So this one goes the other way round, through the Amazon services we lean on, roughly in the order your compile request runs into them\. ### Getting your code as far as our servers Your browser talks to[CloudFront](https://aws.amazon.com/cloudfront/), Amazon’s CDN\. We run two separate CloudFront distributions\. The one in front of`godbolt\.org`mostly just hands requests on to our load balancer, caching what it sensibly can and compressing things on the way back out[2](https://xania.org/202608/how-compiler-explorer-runs-on-aws#fn:brotli)\. The bulky static stuff lives in an S3 bucket behind a second distribution at`static\.ce\-cdn\.net`: the compiled JavaScript, the images, the web fonts\. Much the largest part of that is[Monaco](https://github.com/microsoft/monaco-editor), the editor component out of Visual Studio Code, which is what gives you the syntax highlighting and the squiggly underlines and the rest of it\. It’s a lot of JavaScript to send someone who only wants to look at some assembly, so it’s worth having it cached near them\. Sitting in front of that is[WAF](https://aws.amazon.com/waf/), doing our rate limiting\. Our limits are*very*simple and*very*high, mostly because we used to be stricter and it kept catching C\+\+ trainers: a whole classroom behind a conference’s NAT looks like a single very keen IP address\. We could probably do something cleverer with fingerprinting, but raising the limit was easier and it hasn’t been a problem since\.[3](https://xania.org/202608/how-compiler-explorer-runs-on-aws#fn:bans) ### Rather a lot of fleets behind one load balancer Behind CloudFront is a single[Application Load Balancer](https://aws.amazon.com/elasticloadbalancing/application-load-balancer/)\. It works out from the path which cluster a request is for, and then picks a healthy instance in that cluster to send it to\. Almost everything has no special prefix at all and goes to the production fleet, which is where the overwhelming bulk of the traffic ends up\.`/beta\*`and`/staging\*`go to the beta and staging fleets[4](https://xania.org/202608/how-compiler-explorer-runs-on-aws#fn:envs)\.`/winprod\*`goes to a fleet of Windows instances running[MSVC](https://learn.microsoft.com/en-us/cpp/build/reference/compiling-a-c-cpp-program),`/aarch64prod\*`to[Graviton](https://aws.amazon.com/ec2/graviton/)machines that run ARM code natively rather than under emulation, and`/gpu\*`to machines with real NVIDIA cards in them[5](https://xania.org/202608/how-compiler-explorer-runs-on-aws#fn:gpu)\. Each of those is an[Auto Scaling Group](https://aws.amazon.com/ec2/autoscaling/), and these days each one is doubled up: a blue and a green\. To deploy, we bring up the other colour, wait for it to go healthy, point the load balancer’s target group at it and drain the old one\. If it’s wrong, we point it back\. Before that, deploying meant a rolling restart of the fleet with our`ce`command line tool, one instance at a time\. That worked well enough, but backing out a bad release meant rolling the whole fleet forward again onto the previous version, which is a bit slow when you’ve just broken the site\. The scaling itself is simple: we just try and keep the average CPU load below a threshold\. We’ve kicked around more sophisticated ideas but this one is supported out of the box[6](https://xania.org/202608/how-compiler-explorer-runs-on-aws#fn:queuescale)\. ### Renting the bits nobody else wants The production fleet is almost entirely[spot instances](https://aws.amazon.com/ec2/spot/)– unused EC2 capacity, sold off cheap, on the understanding that you can be evicted at two minutes’ notice\. It’s a 60\-90% saving over on\-demand – quite a lot of money, for us\. We can get away with that because our instances don’t really matter\. Everything that needs to survive lives on the shared filesystem, in S3 or in DynamoDB, so an instance that vanishes is just an instance that gets replaced\. We ask for sixteen different instance types across the`m5`,`m6`,`m7`,`r6`and`i3`/`i4i`families\. The allocation strategy is`price\-capacity\-optimized`– Amazon’s way of saying “pick whichever of these is cheapest and least likely to get yanked”\. In practice we get evicted, the scaling group notices, and a new one turns up\. ### Why the compilers are the hard part We have around 6,000 compiler entries[9](https://xania.org/202608/how-compiler-explorer-runs-on-aws#fn:double)across 93 languages, and we never delete any of them[7](https://xania.org/202608/how-compiler-explorer-runs-on-aws#fn:never)\. Once a compiler version goes up it stays up, so that Stack Overflow answer showing a GCC 4\.8 codegen quirk still compiles today\. It’s our bit against link rot, and it does mean we hoard an awful lot of binaries\. All of that lives on[EFS](https://aws.amazon.com/efs/), Amazon’s elastic NFS, and NFS has latency\. C\-like languages pull in an enormous number of very small header files, so the naive version of this is unusably slow\. Our fix, which sounds daft, is to build a[SquashFS](https://docs.kernel.org/filesystems/squashfs.html)image per compiler, store the image*also*on EFS, and mount it through a loopback device\. The kernel then thinks it’s talking to a local block device and caches blocks properly instead of checking with a server in another building every time[10](https://xania.org/202608/how-compiler-explorer-runs-on-aws#fn:squash)\. That worked, but it meant mounting a couple of thousand images at every boot, which took the best part of a minute and kept the metadata for thousands of filesystems resident in kernel memory\. That cached metadata is a good chunk of why squashfs beats NFS in the first place, so it’s not wasted as such; it’s just that any one instance is only ever going to touch a handful of those compilers, so we were paying for all of it to use a fraction of it\. It took me three years and several abandoned attempts to fix that, and the answer was[CEFS](https://xania.org/202509/cefs): content\-addressed images, packed into bundles of around 20GB, mounted on demand by[autofs](https://www.kernel.org/doc/html/latest/filesystems/autofs.html)the first time something touches the path\. The migration took us from 2,182 images down to 121, and OS startup from 50 seconds to 20\! It’s crept back up to around 883 since, because the nightly builds keep making new images and consolidation only packs down whatever is already there\. A garbage collection while I was writing this turned up 111 images that nothing references any more, about 63GiB’s worth[8](https://xania.org/202608/how-compiler-explorer-runs-on-aws#fn:gc)\. Our storage has come down a lot over the year too, though I’m honestly not sure how much of that is CEFS and how much is me finally deleting the old squash images: ``` Aug/04 01:55 admin-node~ $ df -h Filesystem Size Used Avail Use% Mounted on /dev/nvme0n1p1 97G 20G 77G 21% / fs-db4c8192.efs.us-east-1.amazonaws.com:/ 8.0E 2.2T 8.0E 1% /opt ``` 2\.2T, where a year ago it was 3\.9T\. Still nowhere near 8 exabytes, mind\. ### Building compilers all night, every night We build a pile of compilers from scratch every night: GCC trunk, Clang trunk, and a long tail of experimental branches for reflection, contracts, coroutines and all the other fun stuff\. That’s 94 nightly build jobs at the moment[11](https://xania.org/202608/how-compiler-explorer-runs-on-aws#fn:jobs), up from 73 a year ago and 33 in 2022\. Those run on our own[GitHub Actions](https://github.com/features/actions)runners on[EC2](https://aws.amazon.com/ec2/), spun up on demand with the excellent[terraform\-aws\-github\-runner](https://github.com/github-aws-runners/terraform-aws-github-runner), with all the compiler orchestration on top being ours\. Building trunk LLVM wants a big machine and a decent chunk of time, and GitHub’s hosted runners weren’t up to it when we set this up\. This is the part of the bill that goes up every time somebody asks us for another compiler and we say yes[12](https://xania.org/202608/how-compiler-explorer-runs-on-aws#fn:stale)\. ### The bits that are half\-finished One thing we’ve built but not finished rolling out is what we call the CE Router: a small fleet whose job is to decide*where*a compilation should happen, look the answer up in[DynamoDB](https://aws.amazon.com/dynamodb/)and drop the request on an[SQS](https://aws.amazon.com/sqs/)queue, with the result coming back to your browser over an[API Gateway](https://aws.amazon.com/api-gateway/)WebSocket\. The point is to stop the machine that answers the HTTP request having to be the machine that owns the compiler\. It works, but it isn’t carrying production traffic yet: the load balancer rules that would send compilations through it are still commented out, and we’re migrating gradually\. Your compile today still goes straight to a machine in the prod fleet, the way it always has\. Building it did turn up a good AWS gotcha, though\. API Gateway WebSocket frames top out at 32KiB, and the assembly for almost any non\-trivial program is bigger than that; in the other direction SQS messages cap out at 256KB, which a decent\-sized multi\-file project will go past\. So in both directions, if the thing is too big we shove it in S3 and send a key instead\. ### Everything else A scattering of other services doing one job each: - **[DynamoDB](https://aws.amazon.com/dynamodb/)**holds the short links\. Every`godbolt\.org/z/\.\.\.`you’ve put in a slide deck is a row in a table, and there are a couple of million of them\. - **[S3](https://aws.amazon.com/s3/)**holds the compilers we build, the static assets, the logs, and a daily\-expiring content\-addressable compilation cache, so the second person to compile the same thing gets it for free\. - **[Lambda](https://aws.amazon.com/lambda/)**does the odd jobs: the[Claude Explain](https://xania.org/202505/ai-and-compiler-explorer)backend, the nightly version tracking, and the one that pushes CloudWatch alarms into our Discord so it’s not just my phone buzzing at 3am\. - **[Route 53](https://aws.amazon.com/route53/)**,**[ACM](https://aws.amazon.com/certificate-manager/)**,**[CloudTrail](https://aws.amazon.com/cloudtrail/)**,**[Backup](https://aws.amazon.com/backup/)**and**[SES](https://aws.amazon.com/ses/)**are the plumbing\. I don’t think about them much\. - **[CloudWatch](https://aws.amazon.com/cloudwatch/)**drives the auto\-scaling triggers, though for actually looking at things we run[Grafana](https://grafana.com/),[Prometheus](https://prometheus.io/)and[Loki](https://grafana.com/oss/loki/)\. Those[dashboards are public](https://stats.compiler-explorer.com/)\. Everything is in`us\-east\-1`\. If you’re compiling from Sydney your code goes a long way round, and I’m afraid it’s going to keep doing that: keeping a couple of terabytes of compilers in sync across regions would be an absolute nightmare\. ### Some numbers We log a heavily anonymised JSON record to S3 for every compilation, with a[Glue](https://aws.amazon.com/glue/)table over the top, so while writing this I went and counted them properly:**5,238,210**compilations in July, and**78\.7 million**over the last twelve months\. It’s been sliding gently all year, down about a third from last autumn’s peak\. [](https://xania.org/202608/compilations-per-month.svg) Compilations per month over the last year\. Generated from[this script](https://xania.org/202608/generate_compilation_chart.py)\. The big drop before that, from 2024’s 14 million a month, I can at least half explain: in mid\-2024 we doubled the default delay before we auto\-recompile as you type, from 750ms to 1500ms, and then to 2 seconds[13](https://xania.org/202608/how-compiler-explorer-runs-on-aws#fn:delay)\. If you type continuously, doubling that delay roughly halves the number of compilations you generate, without anybody noticing anything much\. This latest slide isn’t that, though: it started well over a year later\. So I don’t really have a good explanation, and I can’t easily go and find one, because we deliberately[don’t track who you are](https://godbolt.org/#privacy): no cookies, nothing that would let me tell you how many people used the site yesterday\. I’d make that trade again every time, but it does leave me squinting at graphs occasionally\. Since I had the query open, here’s what folks actually compiled in July: ``` id compiles compiler g161 1,386,532 GCC 16.1 (C++) cg161 403,659 GCC 16.1 (C) gsnapshot 298,256 GCC trunk clang_trunk 287,885 Clang trunk g152 239,331 GCC 15.2 clang2210 177,958 Clang 22.1 vcpp_v19_latest_x64 132,186 MSVC r1970 65,380 Rust 1.97 python314 51,104 Python 3.14 ``` That top entry is**26%**of everything – roughly what you’d expect when you’re the default[15](https://xania.org/202608/how-compiler-explorer-runs-on-aws#fn:top10)\. The load balancer sees around 14 million requests a month; our busiest day this year was the 21st of April at 1\.44 million\. The auto\-scaling dealt with that one by itself; I only found out about it while writing this\. All of that costs us somewhere around $3,600 a month on AWS at the moment, which works out at about $0\.0007 per compilation[14](https://xania.org/202608/how-compiler-explorer-runs-on-aws#fn:percomp)\. For the best part of a year we barely paid any of that: AWS’s[open source credits](https://aws.amazon.com/blogs/opensource/aws-promotional-credits-open-source-projects/)covered almost all of it\. Those ran out in the spring, we paid our own way for a few months, and AWS have just renewed them for another year, which is a huge help\. Either way we’re fine for money, thanks to our[Patreon](https://patreon.com/mattgodbolt)supporters, our[GitHub sponsors](https://github.com/sponsors/mattgodbolt)and our[commercial sponsors](https://godbolt.org/#sponsors)\. I did a[full breakdown of the costs](https://xania.org/202506/compiler-explorer-cost-transparency)last year and it’s still roughly the right shape\. [](https://xania.org/202608/compiler-wall-dynamic.svg) The wall of compilers, a year on: 6,157 entries across 93 languages\. Regenerated from[the same script](https://xania.org/202608/generate_compiler_wall.py)as last year's\. ### Things I’d still like to fix Deployment is better than it was but there’s still more hand\-holding in it than I’d like[16](https://xania.org/202608/how-compiler-explorer-runs-on-aws#fn:discovery)\. The CE Router migration has been “nearly done” for a while now… I still wish I’d laid out the NFS directory structure properly at the start instead of growing it one compiler at a time for a decade\. And we moved everything to[Ubuntu 24\.04](https://releases.ubuntu.com/noble/)this year, which was not the tidy little upgrade it sounds like[17](https://xania.org/202608/how-compiler-explorer-runs-on-aws#fn:relr)\. None of this is especially clever: it’s a load balancer, some scaling groups full of cheap spot instances, a network filesystem and a decade of accumulated hacks\. It does keep working, though, and these days it mostly keeps working without anyone having to go and poke it\. I’m pretty happy with that\. ### Thanks As ever, the people who keep this running are the contributors\.[Partouf](https://bsky.app/profile/partouf.bsky.social)\(Patrick Quist\) above all – I really don’t know what CE would do without them – along with the core team and the many, many people who send PRs and file issues\. Thank you all\. Thanks too to AWS, whose[open source credits programme](https://aws.amazon.com/blogs/opensource/aws-promotional-credits-open-source-projects/)paid for the best part of a year of everything described above, and who have just renewed for another year – which is not a small thing for a project our size, and it lets us get on with the interesting problems instead of watching the bill\. Thank you AWS\! And thanks, as always, to our[Patreon](https://patreon.com/mattgodbolt)and[GitHub Sponsors](https://github.com/sponsors/mattgodbolt)supporters and our[commercial sponsors](https://godbolt.org/#sponsors), who cover the rest\. Questions? Complaints? Compilers we’re missing? Drop by our[Discord](https://discord.gg/B5WacA7), or find me on[Bluesky](https://bsky.app/profile/matt.godbolt.org)or[Mastodon](https://hachyderm.io/@mattgodbolt)\. --- ### Disclaimer This article was a collaboration between a human and an[LLM](https://anthropic.com/)\. I set it off to dig through our infrastructure repositories and our AWS account for the current numbers, which turned up several things I had wrong, and worked from that\. It also ran the Athena queries and regenerated the graphs\. The opinions, em\- and en\-dashes, and mistakes are all mine\. Posted at 10:19:00 CDT on 5thAugust 2026\.