A single endpoint is the easy part.
Jared Siegel argues for a self-run LLM proxy because one endpoint can expose client, key, model, status, token, and cache behavior across providers. His concrete test is useful. A cache miss should be traceable to one request instead of guessed from a provider dashboard.[1]
The proxy earns its keep when it preserves a route you can question. Which client sent the request? Which virtual key admitted it? Which model did the client request? Which provider and model handled it? Did the call retry? What did it cost? Which request ID can an operator use to investigate?
Visibility is not a license to store the prompt.
Keep requested and served models separate.
A client can ask for one model alias while the proxy routes to a provider deployment or a different model revision. OpenTelemetry's developing conventions keep gen_ai.request.model, gen_ai.response.model, and gen_ai.provider.name as separate fields. The document also warns that a client library may see a proxy while another provider handles the request.[2]
That difference matters during a regression. If an agent reports a quality drop, the requested alias alone cannot prove which model produced the answer. Record both values when the provider returns them. Keep the proxy route and retry count beside them.
Give every call two handles.
OpenAI returns an x-request-id for troubleshooting and recommends logging it in production. A caller may also send a unique X-Client-Request-Id. The client ID remains useful when a timeout prevents the response header from arriving.[3]
Use one ID at the edge and keep the provider's ID in the same record. The pair lets an operator follow a call from coding tool to proxy to provider without using prompt text as the lookup key.
Metadata still needs a policy.
LiteLLM documents a generated call_id, response headers for model and cost, logging callbacks, and a switch that blocks message and response content while retaining metadata such as spend. Its documentation also allows a per-request redaction override only for keys or teams that receive that permission.[4]
Those are product capabilities, not a complete deployment policy. Write who can read logs. Set retention. Separate client identity from a raw API key. Decide whether tool arguments count as content. Test that redaction remains active on errors, retries, streams, and callback failures.
Build the minimum useful record.
LLM_PROXY_FLIGHT_RECORD
CLIENT_REQUEST_ID=[REQUIRED]
PROVIDER_REQUEST_ID=[WHEN_RETURNED]
CLIENT_ID=[REQUIRED]
VIRTUAL_KEY_ID=[REQUIRED]
REQUESTED_MODEL=[REQUIRED]
SERVED_MODEL=[WHEN_RETURNED]
PROVIDER=[REQUIRED]
STATUS=[REQUIRED]
RETRY_COUNT=[REQUIRED]
INPUT_TOKENS=[WHEN_RETURNED]
OUTPUT_TOKENS=[WHEN_RETURNED]
CACHE_STATUS=[WHEN_RETURNED]
CONTENT_MODE=METADATA_ONLY
RETENTION_DAYS=[REQUIRED]
LOG_READER_GROUP=[REQUIRED]The flight recorder below drafts a logging policy packet. It replaces a loose checklist with one control that joins capture fields, content mode, retention, and an inspection receipt. It does not configure a proxy, validate redaction, or prove that any call was recorded.