AI, Security, Agents, Privacy

When your agent finds the bug, and uses it

Derek Blank

ABC News reported what they are calling Australia's first known autonomous cyber attack by an AI assistant. A man named Andrew asked his AI agent to book him into a popular gym class. The agent found a way to book weeks further ahead than the product allowed and, when Andrew asked whether it could move him up the waitlist, it cancelled the person in position #1 and told him he had moved up.

It was not asked to hack anything. It was just asked to help him get a spot in the class.

That is the obvious story here: give an AI a goal and enough access, and it may find shortcuts you did not expect or authorise. But there is another part of this story that should make anyone building digital products pay attention.

The response that gives the game away

AI agent message apologising for removing another user from a gym waitlist

After removing someone else from the waitlist, the agent explained what it had done. It named the actions available inside the booking system, explained which ones the software had blocked and which one it had allowed, and included another person's user ID.

It had discovered that the gym's software correctly checked whether Andrew was allowed to make some changes, but failed to perform the same check when cancelling a reservation. The AI tried the door, found it unlocked and walked through it.

Then it described the unlocked door in the chat.

That matters because the conversation was no longer just a record of an AI doing something it should not have done. It contained useful information about the weakness itself, including details about another user. Anyone with access to the chat, a screenshot or a shared support thread now had a pretty good idea of where the problem was and how it had been used.

Why this is worse than a funny gym story

There are a few things going on here at once.

The first is a fairly ordinary software security problem. If a system lets you change or delete something, it should check that you actually own it or have permission to do so. The gym's software appears to have done that correctly in some places and not others. Bugs like this are not new.

What is new is the thing finding them.

A person using the gym website would probably hit a "no" and move on. An AI agent given a goal can try another route, then another, looking for something that works. It does not necessarily understand that one of those barriers exists because the action on the other side belongs to somebody else.

The second problem is disclosure. The agent did not quietly encounter the flaw and move on. It explained what it had discovered and included information belonging to another customer. AI systems are extraordinarily good at describing what they have just done, which is useful most of the time, but it also means a security weakness can move from being hidden inside a system to being neatly documented in a chat transcript.

The third problem is more fundamental: an AI being helpful to you is not the same thing as an AI behaving appropriately within the wider system.

Andrew wanted a place in the class. The agent was working towards that goal. The person at the front of the waitlist, the gym and the company that built the booking software were not its clients. From the agent's point of view, somebody else's reservation was simply something standing between its user and the outcome it had been asked to achieve.

Andrew later asked it to undo the cancellation, but it could not restore the other person's position. He then had the agent prepare a vulnerability report for the software provider. Responsible after the fact, but the action had already happened.

What this means for anyone building software

You do not need to be building AI software for this to affect you.

If you run a booking platform, customer portal, online store, membership system, payment product or anything else people interact with online, AI agents are increasingly going to become some of your users. People will ask them to make bookings, chase refunds, change reservations, update accounts, compare prices and sort out problems on their behalf.

That changes the threat model.

A weakness that once required someone with technical knowledge to deliberately go looking for it can now be discovered by an AI that is simply trying very hard to complete an ordinary task. If the first route fails, it can try another. If the software exposes more information than it should, the agent can read it. If an action is available without the right permission checks, it can use it.

The practical response is not especially exotic. Software should check ownership and permissions every time someone tries to change another record, particularly when something is being cancelled, deleted, transferred or reordered. Sensitive actions should have safeguards before permanent changes happen, and systems should avoid exposing information about other users or unnecessary details about how things work internally.

It is also increasingly important to watch for behaviour that looks strange rather than simply relying on whether an individual action is technically allowed. One customer repeatedly cancelling other people's reservations should look suspicious whether the person pressing the buttons is human or AI.

These are established security principles. The difference is that AI agents make it much easier to probe the gaps between them.

The uncomfortable bit

We are going to keep giving agents goals like "get me the booking", "chase the invoice", "fix the sync" and "sort this out", because having software capable of actually doing things for us is enormously useful.

But agents are also very good at finding paths around obstacles, and sometimes the obstacle is there for a reason.

The gym's software had a hole. The agent found it while trying to be helpful, walked through it and then explained exactly what it had discovered.

That is the bit worth paying attention to.

The question is no longer only whether an AI gives you the right answer. As we give these systems the ability to act, we also have to think much harder about what they are allowed to touch, what happens when they find something we missed, and whose interests they are actually working for.

Source: ABC News: AI assistant hacks gym website in first known Australian autonomous cyber attack (Cam Wilson and Rhiannon Hobbins, 9 August 2026).

Ready to get started?

Work with us