Should we move metrics to an observability platform and keep ad hoc queries?

[Aubrey.C]

I'm trying to decide if we should move our main application metrics from our current time-series database to a dedicated observability platform, but I'm worried about losing the ability to run complex ad-hoc queries against raw event data. The platform's pre-built dashboards are great, but I feel like we might be trading flexibility for convenience.

[Tyler_J]

We moved to a dedicated observability platform last quarter. We kept a hot path in the time-series DB but still export the raw events to a data lake for occasional dives. The dashboards are fast, but heavy ad-hoc queries on the raw data are slower and require extra prep. It’s helped with alerts and traces, but the cost, retention rules, and training complexity of maintaining two paths are real tradeoffs.

[DavidL]

We ran a two week pilot with a subset of events. We kept raw events accessible via export to a data lake and used the platform’s dashboards for day to day monitoring. The dashboards were great, but when we tried to answer unusual questions, the lack of direct access to raw event structures slowed us down; we built a couple of pipelines to rehydrate data into a notebook. It worked okay, but the maintenance started to feel heavy.

[Scarlett_M]

Is the real problem the dashboards or the data quality? It sure feels like the events coming in are inconsistent across services, which makes any platform less helpful. Maybe we need better event schema governance before moving tooling.

[Kenneth81]

I tried to rely on the new platform for everything and ended up going back to the old time-series DB for heavy cross service analyses. Latency on those paths doubled. We split use cases by data tier and kept a small sandbox for raw signals, which helps, but it feels like we’re chasing two beasts.