Infrastructure-as-Code (Bicep) for deploying Azure AI Foundry (AIServices) instances behind an Azure API Management gateway. Originally developed for the National Synchrotron Light Source II (NSLS-II) at Brookhaven National Laboratory; forks welcome.
This repository was originally derived from Microsoft's
AzureOpenAI-with-APIM
sample and has since been substantially restructured and extended
(multi-region failover, Anthropic API surface, private networking, custom
TLS, token-usage analytics, IP allow-listing). See LICENSE for
the MIT terms inherited from that project, with an added BNL copyright
notice.
You can call models deployed behind APIM using standard OpenAI and Anthropic Python SDKs.
- Python 3.8+
pip install "openai>=1.66.0" anthropic rich—openai>=1.66.0is required for the v1 Responses API example; earlier versions suffice for Chat Completions only- Set your APIM subscription key:
export AIFAPIM_API_KEY=... - The gateway supports HTTP/2; clients using
httpxneedpip install httpx[h2]andhttp2=Trueto negotiate it.
from openai import AzureOpenAI
import os
endpoint = f"https://{os.environ['AIFAPIM_HOST']}"
deployment_name = "gpt-4.1-mini" # or any deployed model name
api_version = "2024-10-21"
api_key = os.environ["AIFAPIM_API_KEY"]
client = AzureOpenAI(
base_url=f"{endpoint}/deployments/{deployment_name}",
api_key="placeholder",
default_headers={"x-api-key": api_key},
api_version=api_version,
)
response = client.chat.completions.create(
model=deployment_name,
messages=[
{"role": "system", "content": "You are an AI assistant."},
{"role": "user", "content": "What is the National Synchrotron Light Source II?"},
],
)
print(response.choices[0].message.content)Path note: The official Anthropic SDK appends
/v1/messagestobase_urlinternally, so setbase_url=https://{AIFAPIM_HOST}/anthropic. Raw HTTP clients and OpenCode (which do not append/v1) should usehttps://{AIFAPIM_HOST}/anthropic/v1instead.
from anthropic import Anthropic
import os
endpoint = f"https://{os.environ['AIFAPIM_HOST']}"
deployment_name = "claude-sonnet-5-5" # or any deployed Claude model
api_key = os.environ["AIFAPIM_API_KEY"]
client = Anthropic(
base_url=f"{endpoint}/anthropic",
api_key=api_key
)
response = client.messages.create(
model=deployment_name,
max_tokens=800,
system="You are an AI assistant.",
messages=[{"role": "user", "content": "What is the National Synchrotron Light Source II?"}],
)
print(response)The azure-openai-v1-messages-api exposes the OpenAI Responses API
(POST /openai/v1/responses). This uses a different schema to Chat
Completions — input is an array of messages and output is returned in
response.output. Requires openai>=1.66.0.
from openai import OpenAI
import os
endpoint = f"https://{os.environ['AIFAPIM_HOST']}"
deployment_name = "gpt-4.1-mini" # or any deployed model name
api_key = os.environ["AIFAPIM_API_KEY"]
client = OpenAI(
base_url=f"{endpoint}/openai",
api_key="placeholder",
default_headers={"x-api-key": api_key},
)
response = client.responses.create(
model=deployment_name,
input=[
{"role": "system", "content": "You are an AI assistant."},
{"role": "user", "content": "What is the National Synchrotron Light Source II?"},
],
)
print(response.output[0].content[0].text)The Black Forest Labs FLUX models are deployed as DataZoneStandard, so
prompts and images are processed within the US data zone. Images come back
base64-encoded in data[].b64_json, as PNG or JPEG (FLUX-1.1-pro returns
JPEG), so check the first bytes rather than assuming a format. What works
through the gateway, tested on prod:
| Model | Create | Edit |
|---|---|---|
FLUX-1.1-pro |
yes | no |
FLUX.1-Kontext-pro |
yes | yes |
FLUX.2-pro |
yes, provider route | yes, provider route |
- Create:
POST /openai/v1/images/generations?api-version=previewwith a JSON body (model,prompt,size,n).FLUX-1.1-proaccepts a width and height of up to 1440. - Edit:
POST /deployments/FLUX.1-Kontext-pro/images/editswithapi-version=2025-04-01-preview(no other version works) and a multipart form ofimageandprompt. Do not sendn: it is rejected. The v1 route/openai/v1/images/editsfails with "Model not supported with Responses API", so the OpenAI SDK'simages.editdoes not work against it. FLUX.2-prois served by Azure only on Black Forest Labs' provider route, so the gateway has a separate API for it:POST /providers/blackforestlabs/v1/flux-2-pro(the path Azure uses, on the gateway host). Send JSON withmodel(FLUX.2-pro),prompt,width,heightandoutput_format; for editing addinput_image(andinput_image_2and so on, up to eight) as base64, not URLs. The response is synchronous, with the image indata[0].b64_json.api-versiondefaults topreview. A width and height up to 4 MP are accepted by the model; see Black Forest Labs for the other parameters.
import base64
import os
import requests
from openai import OpenAI
endpoint = f"https://{os.environ['AIFAPIM_HOST']}"
headers = {"x-api-key": os.environ["AIFAPIM_API_KEY"]}
client = OpenAI(
base_url=f"{endpoint}/openai/v1/",
api_key="placeholder",
default_headers=headers,
default_query={"api-version": "preview"},
)
result = client.images.generate(
model="FLUX.1-Kontext-pro",
prompt="A calm abstract illustration of teal light beams, no text",
size="1024x1024",
)
image = base64.b64decode(result.data[0].b64_json)
response = requests.post(
f"{endpoint}/deployments/FLUX.1-Kontext-pro/images/edits",
params={"api-version": "2025-04-01-preview"},
headers=headers,
files={"image": ("in.png", image, "image/png")},
data={"prompt": "Change the beams to amber, keep everything else the same"},
timeout=180,
)
response.raise_for_status()
edited = base64.b64decode(response.json()["data"][0]["b64_json"])
response = requests.post(
f"{endpoint}/providers/blackforestlabs/v1/flux-2-pro",
headers=headers,
json={
"model": "FLUX.2-pro",
"prompt": "A calm abstract illustration of teal light beams, no text",
"width": 1024,
"height": 1024,
"output_format": "png",
},
timeout=180,
)
response.raise_for_status()
flux2 = base64.b64decode(response.json()["data"][0]["b64_json"])Image models are billed and limited per image, not per token: the token
limit and token metric policies are skipped for /images/ requests, and the
policies emit an Image Count metric instead (same namespace and dimensions
as the token metrics). They are not listed in GET /catalog, which holds
chat models only. GET /catalog/images lists them, and
GET /catalog/embeddings lists the embedding models. Each image entry gives
the gateway route to use, because the routes differ between model families.
MAI-Image-2.6 and MAI-Image-2.6-Flash use Microsoft's MAI routes, on the
gateway at the path Azure uses: POST /mai/v1/images/generations (JSON with
model, prompt, width, height) and POST /mai/v1/images/edits (multipart
with model, prompt and up to five image files). Both return one PNG in
data[0].b64_json. Width and height are at least 768, with a total of at most
2,359,296 pixels. web_grounding: true is refused with HTTP 400, because it
sends the prompt to Bing. These are GlobalStandard deployments, so prompts
may be processed outside the US.
gpt-image-2.5-flare (fast) and gpt-image-2.5-sunburst (best for precise
edits) use the OpenAI v1 routes, /openai/v1/images/generations and
/openai/v1/images/edits with api-version=preview, like the FLUX models, and
are also GlobalStandard. The gpt-image models are billed by token, so the
quality setting affects the cost.
import base64
import os
import requests
endpoint = f"https://{os.environ['AIFAPIM_HOST']}"
headers = {"x-api-key": os.environ["AIFAPIM_API_KEY"]}
response = requests.post(
f"{endpoint}/mai/v1/images/generations",
headers=headers,
json={
"model": "MAI-Image-2.6",
"prompt": "A calm abstract illustration of teal light beams, no text",
"width": 1024,
"height": 1024,
},
timeout=180,
)
response.raise_for_status()
png = base64.b64decode(response.json()["data"][0]["b64_json"])The examples/opencode.json file contains a ready-to-use
OpenCode configuration that connects to both the
Azure OpenAI and Anthropic APIs through this APIM gateway. Copy it to your
OpenCode config location:
# Per-project (in your project root):
cp examples/opencode.json opencode.json
# Global (user-wide):
cp examples/opencode.json ~/.config/opencode/opencode.jsonSet the required environment variables before starting OpenCode:
export AIFAPIM_HOST=<gateway-hostname>
export AIFAPIM_API_KEY=<your-subscription-key>
opencodeThe config registers two providers; the models arrays below reflect
examples/opencode.json as shipped (the model list from the original
deployment, kept as a realistic example). Edit the models arrays to match
the deployments you provisioned in your aifapim-<env>.bicepparam — only
deployment names that actually exist behind the gateway will resolve.
| Provider key | npm package | Models (as shipped in examples/opencode.json) |
|---|---|---|
aifapim-openapi |
@ai-sdk/openai |
GPT-4.1, GPT-4.1 Mini |
aifapim-anthropic |
@ai-sdk/anthropic |
Claude Sonnet 4.5, Claude Haiku 4.5 |
Both providers authenticate using {env:AIFAPIM_API_KEY} passed as the
x-api-key header — no provider API key is required. The gateway hostname is
read from {env:AIFAPIM_HOST}, making it easy to switch between dev and prod
by changing a single environment variable.
For the full OpenCode configuration reference, see the OpenCode config docs.
See examples/test-apim.py for a full script with Markdown rendering and
more advanced usage.
- Multi-region: Two AI Foundry accounts with private endpoints in separate VNets, peered together.
- Private networking: AI Foundry endpoints are disabled for public access; APIM reaches them via private endpoints and shared private DNS zones.
- Managed Identity auth: APIM authenticates to AI Foundry using its system-assigned managed identity (Azure AI User / Cognitive Services OpenAI User roles).
- Custom domains: APIM gateway uses TLS certificates from Azure Key Vault (RBAC-based), with InCommon CA certificates in the trust store.
- Retry & failover: Automatic retry and failover from primary to secondary region.
- Azure CLI (
az) with Bicep support - A subscription with permissions to create Cognitive Services, APIM, VNets, and role assignments
- For custom domains: a Key Vault (RBAC-enabled) with PFX certificates stored as secrets
aifapim.example.bicepparam is the canonical, in-repo parameter template.
Real *.bicepparam files are gitignored and must be created locally per
environment:
cp aifapim.example.bicepparam aifapim-prod.bicepparam
$EDITOR aifapim-prod.bicepparam # fill in your valuesThe example file is fully commented; each parameter explains its purpose,
valid values, and any gotchas. Pay particular attention to allowedClientIps
(the inbound IP allow-list — RFC 5737 documentation IPs in the example will
not allow real traffic), the regions / modelDeployments arrays, and
catalogJson (the example loads model-catalog.example.json; point it at a
catalog that matches your deployments).
az deployment group create \
--resource-group <your-rg> \
--template-file aifapim.bicep \
--parameters aifapim-prod.bicepparamNote: Do not prefix
.bicepparamfiles with@on the--parametersflag. That syntax is for raw JSON parameter files only and causes the CLI to fail withunrecognized template parameter 'using 'aifapim.bicep'...'. For.bicepparam, pass the path directly.
All parameters are documented inline in
aifapim.example.bicepparam. Copy that file to
aifapim-<env>.bicepparam and customize. The canonical source of truth for
parameter types, defaults, validators, and descriptions is the @description
blocks in aifapim.bicep.
Pay particular attention to:
allowedClientIps— required (@minLength(1)); the inbound IP allow-list applied to all the inference APIs. The example file ships RFC 5737 documentation IPs that cannot route real traffic; replace them.regionsandmodelDeployments— nested arrays; outer indices ofmodelDeploymentsmust align withregions[*].apimProductName— the APIM product the five inference APIs (and the catalog API) are grouped under, and the App Insights custom-metric namespace.serviceProducts— optional extra products for service keys, each{ name, displayName, description, allowedClientIps?, entraAuth? }with the same four APIs.allowedClientIpslocks a product to those callers (a CIDR covers its whole range) on top of the global list.entraAuth({ tenantId, audience, role }) makes the product need a Microsoft Entra ID app-only token as well as the key (see "Authentication model").open: truemakes it the open product for people: no key, a person's Microsoft Entra ID token alone; services still need key and token (see "Open product: Entra token, no key"). SeeAGENTS.md"Products".usageTrace— optional; writes one Application Insights trace per request to the five inference APIs that names the caller. Needed for the open product.
aifapim.bicep # Main deployment orchestrator
aifapim.example.bicepparam # Parameter file template — copy and customize per environment
model-catalog.example.json # Example GET /catalog body, loaded by the example parameter file
llm-token-usage-workbook.bicep # Azure Monitor workbook for LLM token usage analytics
modules/
aif.bicep # AI Foundry (AIServices) account + private endpoint
aif-deployments.bicep # Model deployments on AI Foundry
api-management-private.bicep # APIM instance with custom domains & CA certs
api.bicep # API definitions, backends, product, diagnostics
app-insights.bicep # Application Insights
cognitive-services-role.bicep # Cognitive Services / AI User RBAC for APIM system identity
event-hub.bicep # Event Hub namespace (private endpoint only; not wired to an APIM logger yet)
keyvault-role.bicep # Key Vault RBAC for APIM user-assigned identity
llm-latency-alerts.bicep # Scheduled-query latency alerts on ApiManagementGatewayLogs
log-analytics-workspace.bicep # Log Analytics workspace
network.bicep # APIM subnet
private-dns-zone.bicep # Private DNS zone (AI Foundry endpoints, Event Hub, request log storage)
request-log.bicep # Request body log wrapper (optional)
request-log-storage.bicep # WORM storage account for request bodies
request-log-function.bicep # Loader function app, private subnet, Event Grid, roles
request-log-ampls.bicep # Azure Monitor private link scope and private endpoint
request-log-workspace.bicep # Prompts workspace, custom table, DCE and DCR
role.bicep # Generic role-assignment helper
storage-private-endpoint.bicep # Private endpoint for a storage account
vnet.bicep # Virtual network
apim_policies/
AOAI_Policy-Managed_Identity_with_Retry_MultiRegion.xml # AOAI multi-region with failover
OpenAIv1Messages_Policy-Managed_Identity_with_Retry_MultiRegion.xml # AOAI v1 Messages multi-region with failover
Anthropic_Policy-Managed_Identity_with_Retry_MultiRegion.xml # Anthropic multi-region with failover
BlackForestLabs_Policy-Managed_Identity_with_Retry_MultiRegion.xml # FLUX.2-pro provider route with failover
MAI_Image_Policy-Managed_Identity_with_Retry.xml # MAI image routes, one backend
Catalog_Policy.xml # Static return-response; no backend
api_definitions/
AzureOpenAI_inference_2024-10-21.yaml # AOAI data-plane inference spec; vendored from azure-rest-api-specs
AzureOpenAI_v1_Messages_OpenAPI.json # AOAI v1 Messages OpenAPI spec imported into APIM
AzureAnthropic_OpenAPI.json # Anthropic OpenAPI spec imported into APIM
BlackForestLabs_OpenAPI.json # FLUX.2-pro (Black Forest Labs route) OpenAPI spec
MAI_Image_OpenAPI.json # MAI image OpenAPI spec
Catalog_OpenAPI.json # GET /catalog, /catalog/images and /catalog/embeddings static model lists
resources/ # User-supplied CA PEMs loaded via loadTextContent
# in the bicepparam file. Directory is gitignored
# aside from a .gitkeep; see "Custom Domains & TLS".
examples/
test-apim.py # Smoke-test against APIM using a subscription key
opencode.json # Ready-to-use OpenCode configuration
All policies include:
- IP filtering: Inbound traffic is restricted to entries in the
allowedClientIpsparameter (see your*.bicepparamfile; the canonical template isaifapim.example.bicepparam). Entries are either bare IPv4 addresses or CIDR ranges; CIDR entries are expanded at deploy time viaparseCidr().firstUsable/lastUsable(network and broadcast addresses are excluded). Typical contents are your client network range plus any provider-side egress IPs you need to allow (for example, the Anthropic egress IPs needed for/anthropic/*traffic on Foundry).
The five inference policies (AOAI, OpenAIv1Messages, Anthropic,
BlackForestLabs, MAI_Image) additionally include:
- Managed Identity auth: APIM authenticates to AI Foundry backends using its system-assigned identity
- Retry & failover: Retry on
429and5xxresponses and on backend connection failures (the gateway logs show a reset connection being retried). Failover from the primary backend to the secondary backend in the other region happens only for models deployed in the secondary region (the__SECONDARY_MODELS__sentinel, built frommodelDeployments[0]); other models, such as the Claude models that exist only ineastus2, retry against the primary. Model listing and retrieval requests never fail over: they carry no model name, and each region lists only its own models. A404means the deployment does not exist and is returned to the client unchanged. - Token metrics: Each LLM API policy emits per-request token metrics to
Application Insights via
<llm-emit-token-metric>(or the AOAI-specific<azure-openai-emit-token-metric>variant) with dimensions for API ID, Subscription ID, User ID, Product ID, and Model. The metric namespace is the configuredapimProductName, substituted into the policy XML at deploy time via the__METRIC_NAMESPACE__sentinel. See Token Usage Analytics below. Requests whose path contains/images/skip the token limit and token metric policies (they carry no tokens) and emit anImage Countmetric on success instead.
The catalog policy (Catalog_Policy.xml) uses <return-response> in
<inbound> — no backend is contacted, so managed identity auth, retry, and
token metrics do not apply. The catalog JSON body is baked in at deploy time
via the __CATALOG_JSON__ sentinel (XML-escaped by aifapim.bicep; the
sentinel must appear only in <set-body>, never in a comment).
Authentication is two-hop and intentionally asymmetric:
- APIM → AI Foundry: APIM authenticates to each AI Foundry account using
its system-assigned managed identity, granted
Cognitive Services OpenAI UserandAzure AI Userroles on each account. No keys are stored. - Client → APIM: End-users authenticate with an APIM subscription key
passed via the
x-api-keyheader, gated by theallowedClientIpsIP allow- list described above.
Key and Entra token (optional, per product). The inference API policies
do not check an Entra ID token, so a plain key works as before. A service
product can ask for one on top of the key: give its entry in serviceProducts
an entraAuth setting ({ tenantId, audience, role }). The gateway then builds
that product's policy from apim_policies/Product_EntraAuth_Policy.xml, which
runs <validate-azure-ad-token> and requires:
- both credentials. The key still comes first: a call with no valid key is
refused by the API, and a call with a key but no valid token gets HTTP 401.
The token is sent as
Authorization: Bearer <token>, the key asx-api-key. - a token for this gateway: signed by the tenant, not expired, issued for
audience(for a v2 token, the client ID of the Entra app that represents the gateway) and holding the app rolerole. - an app-only token obtained with a certificate (
idtypisapp,azpacris2), so a person's token or one got with a client secret is refused.
No list of client apps is kept in the gateway: which apps hold the role is set
in Entra ID, so adding a caller is not a gateway deployment. The token is not
tied to the key it is sent with. The key still identifies the caller for the
token limit, the metrics and the request log. On the way to the models APIM
replaces the Authorization header with its managed-identity token, so the
caller's token never reaches them.
A client library that sends its API key as Authorization: Bearer (the OpenAI
SDK does) must replace that header with the Entra token. The tenant, audience
and role are not secrets, but keep the real values out of this public
repository: set them in the environment's own parameter file, and use the
placeholders in aifapim.example.bicepparam.
Without entraAuth nothing changes. Entra ID token validation is not applied
to the user product or to the other service products.
One entry of serviceProducts can set open: true. Callers of that product
send no subscription key, only Authorization: Bearer <token>. The product's
policy, apim_policies/Product_EntraOpen_Policy.xml, is then the only access
check, and accepts people only:
- A person: a delegated token for this gateway whose
scpclaim holdsentraAuth.scopeand whoserolesclaim holdsentraAuth.userRole(an app role assigned to the person, or to a group they are in, in Entra ID).entraAuth.clientApplicationIds(optional) limits which client apps may ask for a person's token. - Not a service principal. An app-only token is refused with HTTP 403
(
KeyRequired). A service calls with its subscription key and its token through a key-and-token product, so switching its key off always stops it. Without the key, or with a key that is not valid, APIM would send it here.
Things to know:
-
It needs all of
tenantId,audience,scopeanduserRole(roleis not used). With less, or with noentraAuth, the product renders a policy that answers every request with HTTP 503, never an inheritance-only policy, which would leave the product open to anyone. -
An API can be in only one open product. The API policies still say a subscription is required; per the APIM documentation a request with no key (or a key that is not valid) is handled in the open product, and a request with a valid key is handled in the key's own product as before.
-
The caller is the token's
oid(the object id of the person or service principal), which is also how personal subscriptions are named. The API policies read the caller from two variables,usageIdandusageName, which are the subscription's id and name for a keyed request and the token'soidand display name for a keyless one. The token limit counter, theSubscription IDandSubscription Namemetric dimensions and the request body log use them, so a person has one limit and one metric series however they authenticate. -
The caller's token is dropped (
Authorizationis deleted) once it has been validated, so it never reaches the models. -
Usage per caller comes from the usage trace (
usageTrace = true): oneAppTracesrow per request with the caller, request id, API, product and model. Join it toApiManagementGatewayLlmLogonCorrelationIdfor tokens (Anthropic completion tokens are not measured for streamed requests, as for keys). A keyless request has no subscription, so the gateway log alone cannot name the caller. The log holds several rows per request (extra rows with zero tokens), so count requests withdcount(RequestId), notcount().AppTraces | where TimeGenerated > ago(1d) and isnotempty(tostring(Properties.CallerType)) | extend RequestId = tostring(Properties.RequestId), Caller = tostring(Properties.CallerId) | join kind=inner (ApiManagementGatewayLlmLog | where TimeGenerated > ago(1d)) on $left.RequestId == $right.CorrelationId | summarize Requests = dcount(RequestId), PromptTokens = sum(PromptTokens), CompletionTokens = sum(CompletionTokens) by Caller, CallerName = tostring(Properties.CallerName), CallerType = tostring(Properties.CallerType)
-
Who may get a token is decided in Entra ID: expose the delegated scope on the gateway API app and a user app role, pre-authorize the client apps, assign the role to the intended groups, and set "Assignment required" on the enterprise application so only they receive a token. The gateway checks the role as well, so widening access later is a role assignment, not a gateway deployment. Nothing here lists users.
Token usage for the LLM APIs published through this gateway is captured in two complementary sinks in the Log Analytics workspace / Application Insights component connected to the APIM service:
ApiManagementGatewayLlmLogtable — populated by APIM's LLM diagnostic setting (apim-genai-gateway-logs, defined inmodules/api-management-private.bicep). Reliable for AOAI / OpenAI v1 Messages prompt and completion tokens. Has gaps for the Anthropic API (see Platform behaviors below).AppMetricstable — populated by APIM's auto-emitted metrics, by the explicit<llm-emit-token-metric>/<azure-openai-emit-token-metric>policies on each API, and by the Anthropic policy's<outbound><emit-metric>block. Use this as the primary source for Anthropic token accounting. All custom-policy metrics emitted by this repo land under the namespace given byapimProductName(the policy XML uses a__METRIC_NAMESPACE__sentinel substituted at deploy time).
| Field | AOAI in LlmLog |
Anthropic in LlmLog |
Anthropic in AppMetrics |
|---|---|---|---|
| Prompt tokens | populated | populated | populated (Prompt Tokens via <llm-emit-token-metric>) |
| Completion tokens | populated only when client sets stream_options.include_usage=true |
always 0 (streaming and non-streaming) | non-streaming: populated (Anthropic Completion Tokens from <outbound> policy). streaming: unmeasurable (SSE body cannot be parsed in <outbound>) |
ModelName / Model |
populated on most streaming rows (~12/15 sample) | always empty (0/194) | populated under Properties.Model |
DeploymentName |
populated | populated | (in Properties.Model) |
Read from AppMetrics, not from ApiManagementGatewayLlmLog. Prompt tokens
land under the unprefixed Prompt Tokens family (emitted via
<llm-emit-token-metric>); completion tokens land under
Anthropic Completion Tokens (emitted via the <outbound> <emit-metric>
block, non-streaming only):
AppMetrics
| where TimeGenerated > ago(7d)
| extend
SubscriptionId = tostring(parse_json(Properties)["Subscription ID"]),
Model = tostring(parse_json(Properties)["Model"]),
ApiId = tostring(parse_json(Properties)["API ID"])
| where ApiId == "anthropic-service-api"
| where Name in ("Prompt Tokens", "Anthropic Completion Tokens")
| summarize Tokens = sum(Sum) by SubscriptionId, Model, Name
| evaluate pivot(Name, sum(Tokens))AppMetrics
| where TimeGenerated > ago(7d)
| where Name in ("Prompt Tokens", "Completion Tokens", "Total Tokens")
| extend
SubscriptionId = tostring(parse_json(Properties)["Subscription ID"]),
Model = tostring(parse_json(Properties)["Model"]),
ApiId = tostring(parse_json(Properties)["API ID"])
| where ApiId in ("azure-openai-service-api", "azure-openai-v1-messages-api")
| summarize Tokens = sum(Sum) by SubscriptionId, Model, Name
| evaluate pivot(Name, sum(Tokens))AppMetrics
| where TimeGenerated > ago(7d)
| extend
SubscriptionId = tostring(parse_json(Properties)["Subscription ID"]),
Model = tostring(parse_json(Properties)["Model"]),
ApiId = tostring(parse_json(Properties)["API ID"])
| extend Kind = case(
Name == "Prompt Tokens", "Prompt",
Name == "Completion Tokens" and ApiId != "anthropic-service-api", "Completion",
Name == "Anthropic Completion Tokens", "Completion",
"Skip")
| where Kind in ("Prompt", "Completion")
| summarize Tokens = sum(Sum) by SubscriptionId, ApiId, Model, Kind
| evaluate pivot(Kind, sum(Tokens))The case clause picks the correct completion-token source per API:
unprefixed Completion Tokens for OpenAI/Llama (where APIM's auto-emit
populates it), and Anthropic Completion Tokens for Anthropic (emitted
explicitly by the Anthropic policy's <outbound> <emit-metric> block).
The unprefixed Completion Tokens value for Anthropic is excluded because
it is always 0.
The maintained workbook lives in the aifapim-config repo
(llm-token-usage-workbook.bicep). Deploy it from there — see
aifapim-config/AGENTS.md §"Workbook deployment" for the full recipe.
The canonical workbook includes:
- Summary tiles (total prompt/completion/total tokens)
- Token usage by API, Model, and APIM Subscription
- Token usage over time (per subscription)
- Cost analytics tiles (per-model and per-subscription cost from Cost Management)
- Optional non-owner read access via an Azure AD group (Monitoring Reader)
This repo retains llm-token-usage-workbook.example.bicep as a reference
snapshot of the workbook as it stood before the move to aifapim-config. It is
not deployed as part of this repo and will diverge from the canonical version as
new features are added. Do not run az deployment group create against the example
file — use the canonical in aifapim-config instead.
Drift warning: the 2026-06 incident (commit
b0c4b86) was caused by deploying a stale workbook copy to prod while a newer version was in dev. Always deploy from the single canonical source inaifapim-config.
The workbook reads from AppMetrics rather than ApiManagementGatewayLlmLog
because the latter has empty ModelName for Anthropic streaming on classic-tier
APIM, and CompletionTokens=0 for all Anthropic rows (streaming and non-streaming).
The following limitations have been verified on this deployment. They appear to be APIM platform behavior on classic tiers; some may be addressed in v2 tiers per Microsoft's AI gateway capabilities documentation, which notes that the Anthropic Messages API is "currently supported in API Management v2 tiers":
- Anthropic
CompletionTokensis always 0 inApiManagementGatewayLlmLogfor both streaming and non-streaming responses. APIM's classic-tier diagnostic does not parse Anthropic Messages API response payloads. The Anthropic policy XML compensates by readingusage.output_tokensfrom the response body in<outbound>and emitting anAnthropic Completion Tokenscustom metric. - Anthropic streaming completion tokens remain unmeasurable. The
<outbound>workaround above is gated on arequestIsStreamflag because SSE-streamed responses cannot be parsed as a single JSON object viacontext.Response.Body.As<JObject>(). Streaming Anthropic rows showCompletion = 0in the workbook. - Anthropic streaming
ModelNameis always empty inApiManagementGatewayLlmLog. UseDeploymentNamefrom the same row, orProperties.ModelfromAppMetrics(the<llm-emit-token-metric>policy on the Anthropic API extracts the model from the request body and emits it as a dimension). - AOAI streaming
ModelNameis empty on a subset of rows, correlating with whether the client requestedstream_options.include_usage=true. - Anthropic prompts are mostly missing. When the request's
systemfield is a list of content blocks (as Claude Code and OpenCode send), theLlmLogrow has the response but no prompt. A plain-stringsystemis logged. See "Request body log" below.
If accurate per-model token accounting in the unified LlmLog table is
critical for downstream tooling, options are:
- File an Azure support case referencing these symptoms and the official Anthropic Messages support note in the AI gateway docs.
- Migrate to a v2 tier (Standard v2 or Premium v2). No in-place migration exists; this requires a parallel deployment with manual config replication.
- Continue using the
AppMetricsqueries above, which already capture the data needed for billing and usage reporting (with the streaming-Anthropic completion-token caveat noted in item 2).
Because of item 5 above, the stack can optionally keep the raw body of every
POST to the five inference APIs. Set requestBodyLogging = true (and
requestLogSubnetIPPrefix, requestLogReaderGroupObjectId) in the parameter
file. The gateway writes each body to a write-once (WORM) blob named by the
request id (CorrelationId in ApiManagementGatewayLogs), and a Flex
Consumption function loads a searchable copy into a separate Log Analytics
workspace (LlmRequestBody_CL).
- It is off by default. It adds a subnet, two storage accounts with private endpoints, a function app, a workspace, a data collection rule and an Azure Monitor private link scope, and the stored bodies contain whatever users send. There is no NAT gateway: BNL policy denies public IP addresses.
- The private link scope changes Azure Monitor DNS for the whole
regions[0]VNet, including the gateway's Application Insights logger (see theAGENTS.mdnote on AMPLS). - The immutability policy is created unlocked and locking is a separate, irreversible step. Blobs move Hot to Cold after 7 days and to Archive after 90, and are never deleted.
- The workspace copy is readable by anyone with subscription-level
Reader. - The function code and operator runbook are in the private
aifapim-configrepository (request-log/).
In addition to the APIM-managed ApiManagementGatewayLogs /
ApiManagementGatewayLlmLog / AppMetrics tables, each AI Foundry account
also writes its own diagnostic logs to the same Log Analytics workspace by
default. This captures traffic that bypasses the gateway
(Entra-ID-authenticated direct callers, internal Foundry portal usage) and
management-plane events (account/deployment configuration changes).
Default-on categories: Audit, AzureOpenAIRequestUsage, and AllMetrics.
The verbose RequestResponse (full request/response bodies) and Trace
(internal traces) categories are off by default; flip them on per-deployment
via aifDiagnosticsEnableRequestResponse=true and
aifDiagnosticsEnableTrace=true when troubleshooting.
Foundry logs land in AzureDiagnostics, distinct from the APIM tables:
AzureDiagnostics
| where TimeGenerated > ago(1h)
| where ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
| summarize count() by Category, Resource, OperationNameAPIM supports custom domain names with TLS certificates from Azure Key Vault. A user-assigned managed identity is created and granted the Key Vault Secrets User role before APIM is deployed, solving the chicken-and-egg problem where APIM needs Key Vault access during creation.
Certificates must be stored as secrets in Key Vault (APIM reads the PFX from the secret URL, not the certificate URL).
Two optional parameters install PEM-encoded CA certificates into APIM's certificate store:
intermediateCaCert→ APIMCertificateAuthoritystorerootCaCert→ APIMRootstore
Both default to empty (skip). Because loadTextContent() runs at bicepparam
compile time, populate them in your *.bicepparam file rather than in the
Bicep template itself:
param intermediateCaCert = loadTextContent('resources/<your-intermediate>.pem')
param rootCaCert = loadTextContent('resources/<your-root>.pem')The bundled resources/ directory is gitignored aside from a .gitkeep —
drop your PEM files there (or any path you prefer, relative to the bicepparam
file) and uncomment the matching lines in aifapim.example.bicepparam. The
module currently installs at most one cert per store; the parameters are
singular strings, not arrays.
With pixi (reproducible conda env; manifest in examples/):
export AIFAPIM_HOST=<gateway-hostname>
export AIFAPIM_API_KEY="<your-key>"
cd examples && pixi run example
# or, from the repo root:
# pixi run --manifest-path examples/pixi.toml exampleOr with a manually-managed Python:
export AIFAPIM_HOST=<gateway-hostname>
export AIFAPIM_API_KEY="<your-key>"
python examples/test-apim.pySee examples/opencode.json and the OpenCode configuration
section above for the full setup.
This repository's contents are largely AI-generated with human
review. See AI_DISCLOSURE.md for details.