TL;DR: AI query logs inside your enterprise tools are capturing real product feedback at the point of friction. The problem is your vendor probably has that data and you don't, and neither side has worked out how to share it.
If this kind of practitioner-level product thinking is useful, the newsletter is where I work through the patterns I'm noticing across enterprise technology. Subscribe if you'd rather hear it from someone still in the seat. Thank you :)
I was pulling together a leadership update recently and needed a consolidated view of every open and in-progress ticket across five separate JIRA boards. Usually that would take me some effort and my morning would be gone. Unfortunately we have the basic Atlassian plan and some work to do around cross project reporting. This time I used Rovo, Atlassian’s AI assistant, and had it ready in minutes.
Rovo landed on what I needed fairly quickly, but the engagement I had with Rovo made me realise something, something I thought was worth sharing.
That prompt revealed how I actually work across those boards. Five boards, each set up by different teams at different times, with their own naming conventions and workflow states. The fact that I needed an AI assistant to get a unified view means the tool wasn’t doing the job I needed it to do, or at least not in the version and config I had.
Now I’m one person trying to do one task. What about others? Every time someone types a question into an AI assistant inside a productivity tool, they’re telling the system something. “Find that document from last week” or “show me the status of X”. Those queries could potentially expose gaps or friction points between what the user expected to do and what the product actually helped them easily achieve.
We’ve been collecting product feedback for years through the usual channels: incident tickets, user surveys, feature request backlogs. They’re all useful, captured through separate systems, from people who were motivated enough to say something. But most users never log a ticket. They just work around the problem and move on.
AI query logs are slightly different. They capture intent at the point of friction, in the flow of the work, from everyone using the tool, whether they’d even bother raising a ticket or not. Those insights and data could be amazingly useful for product teams. A bunch of independent people asking the same question in five different ways tells a product team that a feature either doesn’t exist, has friction or can’t be found. That’s an insight you don’t always get, with data captured in the moment.
And the insight goes beyond individual pain points. AI can reveal system complexity that individual users may not even be aware of. One person querying across five boards doesn’t know that’s unusual. But the system knows differently. It can see that 40 people are running variations of the same workaround, and that the underlying issue could be a design gap, a training gap, or a product level gap.
We may have done similar in user support chatbots, where we mine logs for support insights. But they are a separate, and purposeful engagement. What I think is largely unexplored is the internal tool side, things like Copilot inside Teams or Rovo across Jira and Confluence. The AI assistants now embedded in your platform fall into the same category. They’re all generating query data that many may not be treating as a product feedback channel.
Here’s where it gets a little complicated though. If you’re an enterprise platform manager like me, looking after these systems internally, the query logs from your users could tell you a lot about where your platform setup is causing friction and where people are working around something. But in most cases, you don’t have access to that data in any usable form. Your vendor does. You probably don’t. And even if you could get to it, there’s a governance conversation your organisation hasn’t had yet about whether it’s appropriate to mine what your people are asking the AI.
Now flip it. If you’re the vendor’s product team, you almost certainly have access to aggregated query data across your entire customer base. You’d be silly not to. That’s a real-time feed showing you how customers are actually trying to use your platform versus how you designed it to be used. I’d be surprised if the sharper product teams aren’t already mining it. But even inside the vendor, there’s a question worth asking: is the team building the AI assistant working closely with the broader product team, or are they operating independently? Because if the AI feature team is using query data to improve their own capability but not feeding those insights back to the core product team, the gaps in the underlying product stay invisible to the people who could actually fix them. And the enterprise PM on the other end rarely sees any of it at all.
The person managing the platform day to day would benefit from understanding what their users are asking the AI. The vendor building the platform is probably already using that data. But the two sides aren’t sharing it, and in most cases there’s no mechanism for them to. It’s probably never even been brought up in most conversations. It would be great if our vendors could surface anonymised query patterns back to us. Whether they’d be willing to is a different thing though, because those logs can also reveal where the product isn’t working, and that’s not a conversation vendors want to have openly.
I think most vendors are still in the “adoption metrics” phase, counting queries and tracking how many people interacted with the AI feature this month. That’s fine as a starting point, but it’s a bit like counting how many people walked into a shop without paying attention to what they picked up, put down, and walked out without buying. The vendors who start reading those logs properly are going to learn things about their own product that their roadmap process may not have surfaced.
I don’t have a clean answer on how this should work. The privacy dimension alone makes it more than a product analytics question. But I think there’s a genuine feedback signal sitting in those logs that most organisations haven’t thought to look at yet. The irony is that they’re now giving us direct feedback via the AI chat, and we are unaware of some of the value within it.
~ Pete G