OTEL_SEMCONV_STABILITY_OPT_IN latency buckets too big

[bergur88]

Describe your environment

services are build with docker python:3.10.15-slim and run on k8s
services use
opentelemetry-api==1.27.0
opentelemetry-sdk==1.27.0
opentelemetry-propagator-b3==1.27.0
opentelemetry-exporter-otlp-proto-grpc==1.27.0
opentelemetry-instrumentation-fastapi==0.48b0
opentelemetry-instrumentation-aiohttp-client==0.48b0
opentelemetry-instrumentation-asyncpg==0.48b0
opentelemetry-instrumentation-psycopg==0.48b0
opentelemetry-instrumentation-psycopg2==0.48b0
opentelemetry-instrumentation-requests==0.48b0
opentelemetry-instrumentation-logging==0.48b0
opentelemetry-instrumentation-system-metrics==0.48b0
opentelemetry-instrumentation-grpc==0.48b0

What happened?

I'm using the OTEL_SEMCONV_STABILITY_OPT_IN feature (I'm currently running http/dup ) and am seeing some weird results with http latencies. It seems to me to use the same bucket sizes as the old metrics. Doesn't the buckets need to be smaller since the unit has been changed from milliseconds to seconds, with the lowest bucket being 5 seconds it not particularly useful and most percentiles calculated from my metrics show that p99 for most of my services/paths are 5 seconds which is not very accurate.

nodejs and dotnet overwrite the default buckets with more sane values.

The images show the same metric during the same time for the same labelset as a histogram and the older one being more granular and useful.
sum(rate(http_server_duration_milliseconds_bucket{app="x", environment="dev"}[$__rate_interval])) by (le)
sum(rate(http_server_request_duration_seconds_bucket{app="x", environment="dev"}[$__rate_interval])) by (le)

Steps to Reproduce

set OTEL_SEMCONV_STABILITY_OPT_IN="http/dup"

it can then be visualized in graphana similarly to this:
sum(rate(http_server_duration_milliseconds_bucket{app="x", environment="dev"}[$__rate_interval])) by (le)
sum(rate(http_server_request_duration_seconds_bucket{app="x", environment="dev"}[$__rate_interval])) by (le)

Expected Result

I expected to see the same percentiles for my services/paths using the semantic metrics.

Actual Result

new metrics are scewed towards 5 seconds because of buckets sizes.

Additional context

No response

Would you like to implement a fix?

None

[emdneto]

I think until API supports advisory implementation so we can pass explicit boundaries, you may want to use views to set the aggregation for the metric. As example:

from opentelemetry.metrics import (
    set_meter_provider,
)
import requests
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.view import View, ExplicitBucketHistogramAggregation
from opentelemetry.sdk.metrics.export import (
    ConsoleMetricExporter,
    PeriodicExportingMetricReader,
)

view = View(
    instrument_name="http.client.duration", aggregation=ExplicitBucketHistogramAggregation(boundaries=[0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1, 2.5, 5,7.5, 10,])
)
reader = PeriodicExportingMetricReader(exporter=ConsoleMetricExporter(), export_interval_millis=1000)
provider = MeterProvider(metric_readers=[reader,], views=[view,])
set_meter_provider(provider)

RequestsInstrumentor().instrument()
response = requests.get(url="https://example.com", timeout=2)

[emdneto]

With the merge of open-telemetry/opentelemetry-python#4361 I think we can now solve this