Unlike Copilot, these AI powered assistants can be built and deployed directly within Microsoft 365 to automate repetitive tasks. That got us wondering: how well do they perform in real world use?
Joe, our Business Systems Consultant, put them to the test. They’re not a one size fits all solution, but in the right scenarios, they proved surprisingly effective.
AI Agents go beyond simple chatbots. They don’t just answer questions, they perform tasks, analyse data, and make decisions. Instead of just giving you information, they can actively help with everyday tasks like processing data, automating workflows, and reducing manual admin.
To see what they’re capable of, we built and tested AI Agents within our Microsoft 365 SharePoint. Here are four real world scenarios we explored to evaluate their potential.
One area that can be particularly challenging for team members, especially new staff, is understanding and interpreting organisational policies. To solve this, we built a Policy AI Agent that pulls data from SharePoint and quickly provides answers to common HR and workplace questions.
What it should do: Provide employees with quick, clear, and easy to understand responses to existing policies, even when queries span multiple policies.
What it does do: From our tests it appears to deliver on our goal. The agent reliably responds with accurate, policy-based answers.
Our takeaway: Although we’ve yet to test where it might produce incorrect answers, which could raise concerns about unintended consequences, early signs suggest this agent could be a valuable internal tool and a quick win.
Tracking contract (and/or proposal) details manually can be tedious and prone to error. To streamline this, we created an AI powered Document Comparison Agent that scans and compares signed contracts against customer proposals; or against a template.
What it should do: Identify discrepancies between contracts and proposals (or a base template), helping to find errors or changes that might not be known.
What it does do: It performs well when asked to obtain a specific info about what was include in an example proposal/contact. However, when asked to compare multiple contracts (or proposals), the results were either inaccurate or more commonly incomplete.
Our takeaway: While the concept was promising, the tool proved too unreliable for real world use at this stage. We’ll revisit it in the future.
We developed an AI SOP Agent designed to help employees find technical procedures quickly by blending internet research with internal documentation. The goal was to provide fast, reliable answers to procedural questions.
What it should do: Provide structured, accurate answers to procedural questions by drawing on internal SOPs and credible external sources.
What it does do: While it theoretically reduces time spent searching for documentation, the agent sometimes merges information incorrectly. For example, it presented a Unifi procedure as if it were for 3CX, making it appear as though the data came from our internal SOPs.
Our takeaway: The tool didn’t function as intended. It currently lacks the reliability needed for practical use; but we will likely revisit and test further in the future.
At The I.T. Team, all staff receive an annual wellness grant, but tracking balances manually meant the finance team was fielding the same questions repeatedly and then had to obtain the information manually from multiple sources. To streamline this, we built a Wellness Grant AI Agent using Copilot Studio and Power Automate. Unlike our other Copilot agents, which took just 15 minutes to set up, this one required several hours of work to get right.
If you're interested in our workflow, here's how it works.
Here’s a screenshot of our Wellness Grant AI Agent at work (values shown are for demo purposes only).
What it should do: Allow employees to instantly check their wellness grant balance via Teams, eliminating the need to disrupt the finance team and wait for a manual response.
What it does do: Success! It does what it should! It successfully reduces back-and-forth communication and saves admin time for the finance team.
Our takeaway: This agent delivers a genuine efficiency gain for our team. It works well, though it did take time and effort to get right; the tangible productivity benefits outweighed the upfront work. We will likely implement this and it should open the door to our team to consider other Use Cases once they start seeing it in action.
To create an AI Agent within Microsoft 365, you'll need a standard Microsoft 365 Copilot licence to design, test, and manage it in Copilot Studio (accessible in Teams). This platform provides the tools to build workflows, integrate data sources, and refine responses.
Once published, your AI Agent can be used by your team without requiring additional licences, as long as it operates within Microsoft 365 tools like Copilot, SharePoint, or Teams. However, if you want it to interact with platforms outside that ecosystem, there may be some costs involved.
Based on our testing we believe AI Agents can be a valuable tool for creating efficiency gains in pockets of your organisations, but there are definitely limits to what they can do. While some examples worked very well and have true potential, others were a complete failure.
If your team spends time on repetitive tasks, searching for information, or dealing with admin-heavy processes, an AI Agent could be worth exploring. It’s important to highlight that it’s difficult to gauge how successful an AI Agent will be for a particular use case until you try it out. A hands-on test run is the best way to determine its value.
Want to explore how AI Agents could work for your organisation? Get in touch. we’d love to help you find the right solution.
