Uses the new Microsoft.Extensions.AI preview to compare the relative performance of OpenAI, AzureAI and xAI, using the same client code.
Currently, comparison includes:
- OpenAI: gpt-4o, gpt-4o-mini
- AzureAI: gpt-4o, gpt-4o-mini
- xAI: grok-beta, grok-2-latest
Prompt:
IList<ChatMessage> prompt =
[
new ChatMessage(ChatRole.System, "You are a chatbot inspired by the Hitchhiker's Guide to the Galaxy."),
new ChatMessage(ChatRole.User, "What is the meaning of life, the universe, and everything?"),
];Results:
BenchmarkDotNet v0.14.0, Windows 11 (10.0.22631.4890/23H2/2023Update/SunValley3)
Intel Core i9-10900T CPU 1.90GHz, 1 CPU, 20 logical and 10 physical cores
.NET SDK 9.0.200-preview.0.25057.12
[Host] : .NET 8.0.13 (8.0.1325.6609), X64 RyuJIT AVX2
DefaultJob : .NET 8.0.13 (8.0.1325.6609), X64 RyuJIT AVX2
| Method | Client | Provider | Model | Mean | Error | StdDev | Median |
|---|---|---|---|---|---|---|---|
| Chat | aai-gpt-4o | Azure AI | gpt-4o | 1.724 s | 0.2150 s | 0.6029 s | 1.493 s |
| Chat | aai-gpt-4o-mini | Azure AI | gpt-4o-mini | 1.897 s | 0.2355 s | 0.6407 s | 1.730 s |
| Chat | oai-gpt-4o | OpenAI | gpt-4o | 1.678 s | 0.1188 s | 0.3408 s | 1.660 s |
| Chat | oai-gpt-4o-mini | OpenAI | gpt-4o-mini | 2.835 s | 0.2741 s | 0.7995 s | 2.629 s |
| Chat | xai-grok-2-latest | xAI | grok-2-latest | 1.629 s | 0.1254 s | 0.3676 s | 1.588 s |
| Chat | xai-grok-beta | xAI | grok-beta | 1.678 s | 0.1107 s | 0.3194 s | 1.655 s |
You can trivially update (and optionally send a PR) the results by running the benchmarks locally.
Simply fork the repository, clone it locally and run the .\update.ps1 script from a pwsh prompt.
The only requirement is to have jq installed and available in your PATH.