Building an internal AI engineering stack on the same platform you ship to customers is a powerful way to validate your technology and uncover real-world challenges early. By treating our own team as a demanding enterprise customer, we were able to stress-test our infrastructure at scale, refine our developer experience, and align our product roadmap with practical AI workloads.<\/p>\n
This article walks through how we designed and implemented an internal AI stack capable of handling tens of millions of requests and hundreds of billions of tokens, and what that means for businesses and development teams planning their own AI infrastructure.<\/p>\n
Instead of assembling a patchwork of external AI tools, we made a deliberate choice: build our internal AI engineering stack on the same platform components we offer to customers. This decision was based on three priorities: reliability, scale, and strategic learning.<\/p>\n
By routing more than 20 million AI requests<\/strong> through our own AI Gateway<\/strong>, processing over 241 billion tokens<\/strong>, and supporting more than 3,683 internal users<\/strong>, we put our infrastructure under the kind of pressure usually seen only in production for demanding clients. That forced our engineering teams to solve the same problems our customers face\u2014at scale.<\/p>\n \u201cIf your own teams can\u2019t successfully run mission-critical AI workloads on your platform, your customers probably can\u2019t either.\u201d<\/strong><\/p>\n<\/blockquote>\n Running AI workloads internally across support, product, security, and engineering teams gave us a broad cross-section of use cases: from natural language assistance to code generation, summarization, and internal analytics. Each of these had different latency, reliability, and cost profiles.<\/p>\n This internal pressure shaped our platform roadmap. We quickly learned where we needed better observability, more efficient routing, rate limit controls, and robust fallbacks across multiple AI models.<\/p>\n Our internal stack is built around three core layers: the AI Gateway<\/strong>, inference on a serverless compute layer<\/strong>, and developer tooling and governance<\/strong>. Together, they allow teams to experiment rapidly while maintaining control over performance and costs.<\/p>\n At the heart of the stack is an AI Gateway<\/strong> that all internal AI traffic flows through. This layer is responsible for:<\/p>\n This design allowed us to operate as if we were a large enterprise with multiple business units\u2014each running different AI workloads, but all governed centrally. It also gave us a single place to enforce best practices and monitor usage across the organization.<\/p>\n To actually execute AI workloads, we rely on a serverless, distributed compute layer<\/strong>. Inference runs close to users geographically, reducing latency and enabling fast, interactive AI experiences even as the number of internal users grows.<\/p>\n Because this compute layer scales automatically, there is no need for teams to provision or manage infrastructure. Whether a tool serves ten users or thousands, the same underlying platform handles the increased traffic without manual intervention.<\/p>\n Internal adoption happened quickly. With more than 20 million AI requests<\/strong> and 241 billion tokens processed<\/strong>, capacity and cost management became critical operational concerns. We focused on three main areas: observability, token control, and model strategy.<\/p>\n We built detailed metrics and dashboards on top of the AI Gateway to track:<\/p>\n This visibility allowed product owners and engineering leads to make informed decisions about which features to optimize or scale back, and where to improve prompts or model selection to reduce unnecessary token usage.<\/p>\n With hundreds of billions of tokens in play, token efficiency became a first-class design constraint. We implemented:<\/p>\n This balance between flexibility and governance is crucial for any business deploying AI at scale\u2014especially when multiple teams are deploying their own tools independently.<\/p>\n Over 3,683 internal users<\/strong> rely on the AI stack for day-to-day work. Supporting this number of users required not just infrastructure, but also strong developer experience<\/strong> and operational discipline<\/strong>.<\/p>\n Developers needed to build AI-powered features without wrestling with infrastructure. To make that possible, we provided:<\/p>\n As a result, teams could ship prototypes in days, not weeks, and standardize on shared patterns that are easier to maintain and secure.<\/p>\n As AI usage expanded, governance became essential. We introduced policies and controls for:<\/p>\n These measures are particularly important for businesses in regulated industries or those handling sensitive customer data. The same architecture can be extended to enforce data residency, encryption standards, and integration with existing identity and access management systems.<\/p>\n For business owners and developers, the lessons from building this internal AI stack are directly applicable to modern web hosting and application architectures. AI workloads are quickly becoming a standard part of web applications, and they demand:<\/p>\n Whether you are enhancing an existing SaaS platform with AI features or building a new AI-first product, treating AI infrastructure as part of your core web platform\u2014not as a bolt-on service\u2014will position you for long-term scalability.<\/p>\n If you are planning or revising your AI stack, consider the following steps:<\/p>\n By building our internal AI engineering stack on the same platform we ship, we validated our architecture under demanding, real-world conditions: millions of requests, hundreds of billions of tokens, and thousands of users. This approach forced us to design for scalability, governance, and cost-efficiency from day one.<\/p>\n For organizations looking to integrate AI into their products and internal tools, the core principles remain consistent: centralize control with an AI gateway, leverage scalable hosting and compute, prioritize observability, and treat internal teams as your first\u2014and most honest\u2014customers.<\/p>\n\n
Aligning Product Development With Real AI Workloads<\/h3>\n
\nArchitecting the Internal AI Engineering Stack<\/h2>\n
1. Centralizing Traffic Through an AI Gateway<\/h3>\n
\n
2. Running Inference on a Serverless Edge Platform<\/h3>\n
\n
\nScaling to Millions of Requests and Billions of Tokens<\/h2>\n
Observability and Usage Analytics<\/h3>\n
\n
Controlling Token Consumption and Cost<\/h3>\n
\n
\nServing Thousands of Internal Users Effectively<\/h2>\n
Streamlined Developer Experience<\/h3>\n
\n
Governance, Security, and Compliance<\/h3>\n
\n
\nWhat This Means for Your AI Infrastructure Strategy<\/h2>\n
\n
Practical Steps to Get Started<\/h3>\n
\n
\nConclusion<\/h2>\n
\n