Skip to main content

Arize AI

AI Observability and Evaluation Platform

tip

This is community maintained, Please make an issue if you run into a bug https://github.com/BerriAI/litellm

Pre-Requisites​

Make an account on Arize AI

Quick Start​

Use just 2 lines of code, to instantly log your responses across all providers with arize

You can also use the instrumentor option instead of the callback, which you can find here.

litellm.callbacks = ["arize"]

import litellm
import os

os.environ["ARIZE_SPACE_KEY"] = ""
os.environ["ARIZE_API_KEY"] = ""

# LLM API Keys
os.environ['OPENAI_API_KEY']=""

# set arize as a callback, litellm will send the data to arize
litellm.callbacks = ["arize"]
 
# openai call
response = litellm.completion(
  model="gpt-3.5-turbo",
  messages=[
    {"role": "user", "content": "Hi 👋 - i'm openai"}
  ]
)

Using with LiteLLM Proxy​

  1. Setup config.yaml
model_list:
  - model_name: gpt-4
    litellm_params:
      model: openai/fake
      api_key: fake-key
      api_base: https://exampleopenaiendpoint-production.up.railway.app/

litellm_settings:
  callbacks: ["arize"]

general_settings:
  master_key: "sk-1234" # can also be set as an environment variable

environment_variables:
    ARIZE_SPACE_KEY: "d0*****"
    ARIZE_API_KEY: "141a****"
    ARIZE_ENDPOINT: "https://otlp.arize.com/v1" # OPTIONAL - your custom arize GRPC api endpoint
    ARIZE_HTTP_ENDPOINT: "https://otlp.arize.com/v1" # OPTIONAL - your custom arize HTTP api endpoint. Set either this or ARIZE_ENDPOINT or Neither (defaults to https://otlp.arize.com/v1 on grpc)
  1. Start the proxy
litellm --config config.yaml
  1. Test it!
curl -X POST 'http://0.0.0.0:4000/chat/completions' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer sk-1234' \
-d '{ "model": "gpt-4", "messages": [{"role": "user", "content": "Hi 👋 - i'm openai"}]}'

Pass Arize Space/Key per-request​

Supported parameters:

  • arize_api_key
  • arize_space_key
import litellm
import os

# LLM API Keys
os.environ['OPENAI_API_KEY']=""

# set arize as a callback, litellm will send the data to arize
litellm.callbacks = ["arize"]
 
# openai call
response = litellm.completion(
  model="gpt-3.5-turbo",
  messages=[
    {"role": "user", "content": "Hi 👋 - i'm openai"}
  ],
  arize_api_key=os.getenv("ARIZE_SPACE_2_API_KEY"),
  arize_space_key=os.getenv("ARIZE_SPACE_2_KEY"),
)

Support & Talk to Founders​