So get this, folks. Microsoft is taking a big hit with its GitHub Copilot services. I’m talking about losing up to 80 bucks a month per user! That’s no chump change.
According to the Wall Street Journal, a “person familiar with the figures” spilled the beans. Here’s the deal: while Microsoft charges users 10 dollars a month for Copilot, they’re actually losing 20 bucks per month on average. And for those heavy users, it gets even worse. They’re costing Microsoft a whopping 80 dollars every 30 days!
Now, in case you’re not familiar, GitHub Copilot is an AI-powered tool that helps programmers write and debug code. Basically, it suggests chunks of code as you type, making your life a whole lot easier. It’s built on OpenAI’s large language models (LLMs), and it got an upgrade earlier this year. They’re using OpenAI’s GPT-3.5 and GPT-4 models now.
We’ve reached out to Microsoft for comment on the cost of running these AI models. If we hear anything back, we’ll fill you in.
Now, running products at a loss isn’t anything new in the tech industry. It’s a strategic move to build a loyal user base before jacking up the prices. Just look at Microsoft’s Xbox games console line. They sell it below cost and make up for it with software sales and content purchases.
And the same logic applies to AI. Microsoft is going all-in on AI, investing heavily to gain an edge in the market.
Now, here’s the thing. Training and running these large language models is not cheap. We’re talking about some expensive hardware. I’m talking about Nvidia’s H100 accelerators, which cost around 30,000 dollars apiece. And get this, I’ve seen them going for 40,000 bucks on eBay!
But that’s not all. Microsoft is using tens of thousands of Nvidia A100s and H100s. So not only do they have to shell out for the hardware, but those servers are gobbling up electricity like there’s no tomorrow.
We don’t have the exact numbers on how much it costs to keep Copilot running, but OpenAI’s CEO, Sam Altman, did say that training GPT-4, their most advanced language model, cost over 100 million dollars!
Now, one way that Big Tech tries to manage the cost of AI is by developing custom accelerators. You’ve got Google’s Tensor Processing Unit and Amazon’s Trainium and Inferentia silicon. And now, if the rumors are to be believed, Microsoft might be getting in on the action with their own custom AI accelerator.
And get this, OpenAI is also considering developing their own custom processor for their machine learning workloads!
So, as it turns out, Microsoft’s current generative AI workloads are running on GPUs. According to Karl Freund, an analyst at Cambrian AI, these models require GPUs due to their latency and bandwidth needs. CPUs just don’t cut it.
And let me tell you, these AI models benefit from large quantities of high-bandwidth memory. They need all that memory to handle the massive amount of parameters in the models. For example, OpenAI’s GPT-3 model, with a mind-blowing 175 billion parameters, may require multiple GPUs per instance!
But it’s important to note that GitHub Copilot isn’t your average chatbot. No, it’s all about code. And specialized models like this usually require fewer parameters, which means less memory and fewer GPUs.
Now, as AI providers figure out the economics of scale, we could see higher prices for these AI features. Just look at Otter.AI, an audio transcription service that’s had its fair share of price hikes and consumption limits over the years.
And guess what? Microsoft and Google are planning to charge a $30 premium on top of their regular subscription plans to unlock gen-AI functionality. They know people are willing to pay for the good stuff!
So don’t be surprised if Microsoft raises the price of GitHub Copilot once it’s convinced enough customers of its value.