Skip to content

Azure Application Insights: internal tools

Idun Cockpit reads the usage of an internal AI tool, for example an assistant built on Azure OpenAI, from the traces it already sends to Azure Application Insights. There is nothing to develop: the Cockpit starts from an export of these traces. It is one of the sources of the real AI usage audit.

  • A tool instrumented with OpenTelemetry, following the OpenInference conventions, whose traces reach Application Insights.
  • Someone who can run a query on the Application Insights resource.
  1. Check the instrumentation

    The traces must carry the user’s question, the answer, the model deployment and the tokens. If they do not, we look at it together at the start.

  2. Export the traces

    Run the query we give you, over the chosen period, and export its result as CSV from the Azure portal, or as JSON.

  3. Send us the export

    Send the file through the channel agreed at the start of the audit.

  • Measured: the question, the answer, the model deployment, tokens, duration, searches and tools called.
  • Not available: the price of each request, which Azure OpenAI does not provide for a private deployment, and the user’s department, absent from the traces.

Often as an identifier already hashed by the application. The Cockpit treats it as a pseudonym, with no address or team.

Which internal tools can be connected this way?

Section titled “Which internal tools can be connected this way?”

Any internal assistant or agent whose traces reach Application Insights, for example an assistant built on Azure OpenAI. What counts is the tool you administer, not the model it calls.

What does the Cockpit draw from these traces?

Section titled “What does the Cockpit draw from these traces?”

The same readings as for your other tools: the use cases, the adoption and what the tool consumes in tokens. The internal tool reads in the same view as your off-the-shelf assistants.

To see what the tool shows on your own tools, book a call.