Azure Application Insights: internal tools
Idun Cockpit reads the usage of an internal AI tool, for example an assistant built on Azure OpenAI, from the traces it already sends to Azure Application Insights. There is nothing to develop: the Cockpit starts from an export of these traces. It is one of the sources of the real AI usage audit.
What you need
Section titled “What you need”- A tool instrumented with OpenTelemetry, following the OpenInference conventions, whose traces reach Application Insights.
- Someone who can run a query on the Application Insights resource.
The steps
Section titled “The steps”-
Check the instrumentation
The traces must carry the user’s question, the answer, the model deployment and the tokens. If they do not, we look at it together at the start.
-
Export the traces
Run the query we give you, over the chosen period, and export its result as CSV from the Azure portal, or as JSON.
-
Send us the export
Send the file through the channel agreed at the start of the audit.
What is measured, and what is not
Section titled “What is measured, and what is not”- Measured: the question, the answer, the model deployment, tokens, duration, searches and tools called.
- Not available: the price of each request, which Azure OpenAI does not provide for a private deployment, and the user’s department, absent from the traces.
Frequently asked questions
Section titled “Frequently asked questions”Is the users’ identity in the traces?
Section titled “Is the users’ identity in the traces?”Often as an identifier already hashed by the application. The Cockpit treats it as a pseudonym, with no address or team.
Which internal tools can be connected this way?
Section titled “Which internal tools can be connected this way?”Any internal assistant or agent whose traces reach Application Insights, for example an assistant built on Azure OpenAI. What counts is the tool you administer, not the model it calls.
What does the Cockpit draw from these traces?
Section titled “What does the Cockpit draw from these traces?”The same readings as for your other tools: the use cases, the adoption and what the tool consumes in tokens. The internal tool reads in the same view as your off-the-shelf assistants.
To see what the tool shows on your own tools, book a call.