Mosaic AI Gateway: Rate Limits, Usage Tables and Inference Tables, with Queries

What this deep dive adds

Chapter 54 of the course names the gateway's three tracking surfaces and what each one answers. This post holds the detail: how to turn them on in code, every rate-limit level and the rules for which limit wins, the columns of the usage and inference tables, and the SQL that turns them into cost attribution and an error report.

Tracking with AI Gateway is a Section 6 (Evaluation and Monitoring) objective in the 18 March 2026 exam guide. What a single endpoint logs on its own is Chapter 53 of the course.

Turning the features on

Each feature is a checkbox in the endpoint's AI Gateway section, or one call through the Databricks SDK. Configuring an endpoint needs CAN MANAGE on it:

from databricks.sdk import WorkspaceClient
from databricks.sdk.service.serving import (
    AiGatewayInferenceTableConfig, AiGatewayRateLimit,
    AiGatewayRateLimitKey, AiGatewayRateLimitRenewalPeriod,
    AiGatewayUsageTrackingConfig,
)

w = WorkspaceClient()
w.serving_endpoints.put_ai_gateway(
    name="support-llm",
    usage_tracking_config=AiGatewayUsageTrackingConfig(enabled=True),
    inference_table_config=AiGatewayInferenceTableConfig(
        enabled=True, catalog_name="main", schema_name="ai_logs",
        table_name_prefix="support_llm"),
    rate_limits=[
        AiGatewayRateLimit(key=AiGatewayRateLimitKey.ENDPOINT, calls=1000,
                           renewal_period=AiGatewayRateLimitRenewalPeriod.MINUTE),
        AiGatewayRateLimit(key=AiGatewayRateLimitKey.USER, calls=60,
                           renewal_period=AiGatewayRateLimitRenewalPeriod.MINUTE),
        AiGatewayRateLimit(key=AiGatewayRateLimitKey.SERVICE_PRINCIPAL,
                           principal="nightly-batch-sp", tokens=200000,
                           renewal_period=AiGatewayRateLimitRenewalPeriod.MINUTE),
    ],
)

The inference table needs CREATE TABLE in the target schema. Changes to the gateway configuration take effect in about 20 to 40 seconds. To stop people bypassing limits by creating their own endpoint, the docs advise keeping endpoint creation and CAN MANAGE with admins and granting everyone else query access only.

Built against the databricks-sdk 0.162 classes, whose objects were constructed and serialised locally; the call itself needs a workspace. The SDK also accepts guardrails and fallback_config. Requirements from the Databricks page "Configure governance on model serving endpoints (legacy)", updated 16 October 2026.

Rate limits: levels and which one wins

• Calls or tokens. Each limit is queries per minute (QPM) or tokens per minute (TPM). When both apply to the same principal, the more restrictive one is enforced. TPM limits are not available on endpoints that serve custom models or agents.
• Several groups. A user in several groups is limited only after exceeding all of those groups' QPM limits, or all of their TPM limits.
• Defaults and caps. No limits exist until you set them; an endpoint takes at most 20 limits, of which at most 5 are group limits, and limits apply only to callers who have permission to query.

Usage tracking: who spent what

Usage tracking shares two system tables. system.serving.endpoint_usage has one row per request: requester (the user or service principal), request_time, status_code, input_token_count, output_token_count, character counts, request_streaming, a usage_context map and served_entity_id. system.serving.served_entities describes each served model, including endpoint_name. Only account admins can query them by default, even though an endpoint manager turns tracking on.

-- Tokens per caller per endpoint, last 7 days
SELECT se.endpoint_name,
       eu.requester,
       COUNT(*)                   AS requests,
       SUM(eu.input_token_count)  AS input_tokens,
       SUM(eu.output_token_count) AS output_tokens
FROM system.serving.endpoint_usage AS eu
JOIN system.serving.served_entities AS se
  ON eu.served_entity_id = se.served_entity_id
WHERE eu.request_time >= current_date() - INTERVAL 7 DAYS
GROUP BY se.endpoint_name, eu.requester
ORDER BY output_tokens DESC;

When one service principal calls on behalf of many customers or projects, requester alone cannot split the bill. Callers of external model endpoints can pass a usage_context map with the request (up to 10 KiB), for example {"project": "returns-bot"}, and group on it with eu.usage_context['project']. When a model does not return token counts, they are estimated as (text length + 1) / 4. Rows typically arrive within an hour.

Inference tables: what was said

The inference table records content, not cost: a table ending in _payload in the chosen schema, with request_date, request_time, databricks_request_id, status_code, execution_duration_ms, the raw request and response JSON, requester, served_entity_id and logging_error_codes.

-- Daily errors and model latency for one endpoint
SELECT request_date,
       COUNT(*)                                    AS requests,
       SUM(CASE WHEN status_code >= 400 THEN 1 ELSE 0 END) AS errors,
       percentile(execution_duration_ms, 0.95)     AS p95_ms
FROM main.ai_logs.support_llm_payload
GROUP BY request_date
ORDER BY request_date DESC;

• Payloads over 1 MiB are not logged in full; logging_error_codes shows MAX_REQUEST_SIZE_EXCEEDED or MAX_RESPONSE_SIZE_EXCEEDED.
• Streaming responses are logged with the chunks aggregated into one response.
• Latency of the log: rows appear within about an hour. Renaming the table, changing its schema or deleting it can stop logging.
• execution_duration_ms is the model's inference time only, without network overhead.

Schemas, limits and behaviour from the Databricks pages "Configure governance on model serving endpoints (legacy)" and "Log requests and responses for serving endpoints (legacy)". The queries are written to those schemas and the join the docs give; they were not run, because this environment has no workspace.

Two more gateway features

The same section configures AI Guardrails (Public Preview): a safety filter, and PII detection that blocks or masks names, addresses and card numbers in requests and responses (output guardrails do not apply to embeddings models or streaming). For endpoints serving external models, fallbacks retry a request that returns 429 or a 5XX error on the next served model, up to two fallbacks, and only the first successful or last failed attempt is logged.

What this means for the exam

Distractors swap the surfaces: an inference table to stop abuse, or a rate limit to explain a bad answer.

FAQ