Since the release of OpenAI’s GPT-3.5, Agio has rapidly accelerated and expanded the delivery of Artificial Intelligence (AI) products to our clients and internal teams. This acceleration is due to the scalable and maintainable tech stack we have assembled in addition to the power of Large Language Models (LLMs). This tech stack enables us to continually experiment with new ideas and when those ideas prove successful, we can rapidly scale our operations. 

This post covers why Agio has chosen OpenAI, FastAPI, and Azure Functions as the main pillars of our tech stack for developing AI products. Additionally, a few small code snippets are included to show how to quickly get these tools working together. 

Why Azure Functions? 

Azure Functions is a serverless compute platform providing a scalable and developer-friendly environment that supports numerous input bindings such as HTTP and event triggers. 

Below explains why to use Azure Functions. 

1. Cost Efficiency and Scalability

Azure Functions aid in lowering resource utilization during periods of inactivity, contributing to cost savings. This cost-efficient approach is particularly advantageous when dealing with varying workloads. 

For example, at Agio, we often have many different APIs in flight at the same time in various stages of maturity from prototype to full production. Azure Functions provide us with a simple way to scale down to zero when a resource is not in use. However, when we encounter a particularly successful use case, we can easily accelerate that prototype to production and scale to handle a high volume of requests. Many of our AI use cases have highly variable volume and Azure Functions are a great fit for this. Azure Functions are adaptable, which enables us to seamlessly scale our resources to match the demand. 

2. Maintainability

The serverless architecture with Azure Functions means developers can focus on writing code without worrying about server provisioning and maintenance.  

This reduces overhead and allows developers to concentrate on AI model development and application code. We’re a relatively lean development team so avoiding the burden of managing servers and clusters makes a real difference. 

3. Monitoring and Debugging

Azure Functions offers easy integration with Application Insights for logging and monitoring metrics. Azure Functions sets up a log handler on the root logger that pushes logs to Application Insights. We are able to centralize our logs to Application Insights (and a single log analytics workspace underlying App Insights) giving us high visibility into the various AI-based APIs. 

See also  Insights from Hedge Fund CTOs: Cracking the Code of Hybrid Workforce Management

4. Easy to Run Locally

Using the Azure Function Core Tools, you can easily run an Azure Function locally. This is very helpful for prototyping, testing and demos. 

5. Option to Expand to an Event-Driven Approach

Azure Functions offers an easy way to swap the trigger on our Azure Functions from HTTP trigger to Event Hub trigger with minimal effort.

For example, at Agio, we are currently focused on developing APIs that our ticketing and monitoring platforms can call synchronously or asynchronously. In the future, we expect to trigger some of these AI products using event-driven approaches.  

Why FastAPI? 

FastAPI is a Python web framework for constructing RESTful APIs. 

1. Great Integration with Azure Functions

Routing, JSON parsing and validation are all handed off to FastAPI with just a few snippets of code and a tweak to function.json settings. 

The following code serves as the entry point for an Azure Function triggered by HTTP. It delegates the request handling to FastAPI:

```python 
from azure.functions import AsgiMiddleware, Context, HttpRequest, HttpResponse 
from fastapi import FastAPI 

app = FastAPI() 
# routes omitted, would be attached to this app 

async def main(req: HttpRequest, context: Context) -> HttpResponse: 
    return await AsgiMiddleware(app).handle_async(req, context) 
``` 

Azure Functions function.json has to be modified to delegate routing to FastAPI (“route” is the key part here). 

```json 
{ 
  "scriptFile": "__init__.py", 
  "bindings": [ 
    { 
      "authLevel": "function", 
      "type": "httpTrigger", 
      "direction": "in", 
      "name": "req", 
      "methods": [ 
        "get", 
        "post" 
      ], 
      "route": "{*route}" 
    }, 
    { 
      "type": "http", 
      "direction": "out", 
      "name": "$return" 
    } 
  ] 
} 
``` 

2. Easy JSON Validation with Pydantic

FastAPI utilizes Pydantic to provide a way to define input and output schemas that are automatically validated by FastAPI and provide very helpful built-in error responses when fields are missing or fail validation. 

```python 
from pydantic import BaseModel 

class Input(BaseModel): 
    short_description: str 
    description: str 
  
class Output(BaseModel): 
    summary: str 

@app.post("/summarize", response_model=Output) 
async def summarize( 
    request_input: Input 
) -> Output: 
    ... 
``` 

3. OpenAPI Spec Generation

See also  Modernize Your Virtual Desktop Infrastructure for Better…Everything

At Agio, we utilize FastAPI auto-generation of OpenAPI spec and publish these as part of our CI/CD process to Azure API Management. This supports handoff of these APIs to integrate with our ticketing platform (ServiceNow). 

```python 
from fastapi import FastAPI 

app = FastAPI() 
# routes omitted, would be attached to this app 
json_spec = app.openapi() 
``` 

4. FastAPI Dependency Management

FastAPI provides a great dependency injection framework, so each route gets access to the dependencies it needs. We use this heavily to provide us with access to instances of service classes we have written, credentials like OpenAI keys, database access, etc. 

In the example below get_summarize_service could provide an iniitialized version of a service (with OpenAI credentials) that calls OpenAI: 

```python 
@lru_cache 
def get_summarize_service(): 
    openai_key = fetch_creds() 
    return MyService(openai_key=openai_key) 
  
@app.post("/summarize", response_model=Output) 
async def summarize( 
    request_input: Input, 
    summarize_service: Annotated[deps.Summarize, Depends(get_summarize_service)], 
) -> Output: 
    ... 
``` 

Using OpenAI with Azure Functions and FastAPI 

OpenAI API calls have high and variable response times. These high latencies make it even more important to leverage asyncio in Python to avoid blocking while waiting for OpenAI to respond. Both FastAPI and Azure Functions support asyncio. 

When making calls to OpenAI we leverage the async version of the Chat Completion API (acreate): 

```python 
import openai 

async def prompt(prompt_txt: str): 
  response = await openai.ChatCompletion.acreate( 
      messages=[ 
          { 
              "role": "system", 
              "content": "metaprompt here" 
          }, 
          { 
              "role": "user", 
              "content": "user prompt here" 
          }, 
      ], 
  ) 
``` 

When an API needs to make several OpenAI calls, where it makes sense we make them in parallel using Python asyncio functions like “gather.” 

```python 
import asyncio 

result1, result2 = await asyncio.gather(prompt("prompt1"), prompt("prompt2")) 

``` 

Final Word 

All-in-all, the combination of OpenAI, Azure Functions, and FastAPI has enabled a single development team at Agio to build a wide range of case and incident enrichment APIs in a brief period this past year.

If you have questions about anything you read here, please feel free to contact us.

Learn More