Privacy risk grows as AI used without clear guardrails – operations expert details serious consequences from simple use

Writing from his experience as a director of operations and development for a UK based home care provider, Phil Tinsley raises a serious concern for the aged care sector the world over, are AI platforms fully understood? And are staff using them without understanding the full risks? Tinsley raises the growing need for management to be across AI use within organisations, and to respond with risk mitigating guardrails. Understanding, clarity and policy shaping must be on the front-foot of keeping pace with technology, as the consequences, even unintended, could result in serious breaches of compliance under national rules on privacy, data and care.

Innocuous start

Throughout his career Tinsley has heard comments shared between colleagues to help improve the workflows of care. Often these comments are administrative and innocuous, trying different techniques of note-taking, or changing approaches to organising. Across the board, the intent is to help, improve and streamline efficiency, in a nutshell, positive inclinations to land efficiency across a workforce and organisation.

What has changed with AI however, is that the efficiencies that come from AI use may have serious consequences resulting in compliance failings, and data-breaches. Where once a suggestion to use a technology to streamline operations may have resulted in reduced administrative time and improved care, may now result in penalties and danger for seniors and the privacy of their health data.

“I only use AI to polish my care notes”, is a phrase that Tinsley hears more and more.

He understands that for most of management, this sounds harmless, perhaps even to be encouraged. For many leaders, he has seen the attitude towards staff using AI, copying care notes into software like ChatGPT, to be appropriate. To ‘just tidy up the spelling’, or to quickly ‘go over grammar’ before being entered formally into the system. For many, this has seemed harmless. But he raises a warning: looking beneath the surface, this is not the case.

The substance of a care note

Tinsley frames the concern surrounding care notes entering into external and third party AI systems, on the substance of the information found within care notes.

Care notes, he assesses, are key, factual written histories of what happened during a visit, shift or interaction with a client. While in many cases they are written without clear names attached, usually using first names or initials, still they contain pertinent, personal and detailed private details.

Tinsley reminds that they contain details of  personal behaviours, patterns of mood, medication use and responses, intimate and personal circumstances and sensitive family dynamics. Over time, Tinsley highlights, it is very possible a pattern builds of who a person is, who their family is and intimate details of their medical, health and care history.

As a staff member copies over, note after note, routinely with the same initial or first name, across varied sessions, even days, weeks or months apart, AI systems may and in some instances have been proven to be building a startlingly detailed account of the individual that is the client, the services they receive and more.

Whether from AI software conversation memory, inference made possible from extensive context provided, the steady build up of facts and detail can result in an outcome where a client’s privacy has been breached by a staff member, and by extension, an organisation. What may have started out as a hope to improve language formality and grammar can now become, Tinsley indicates, a situation resembling a breach of processing personal health data. With this could come the compliance and greater penalties under national and state law.

Data destination

Tinsley notes that the majority of consumer AI tools store the conversation data input on to their systems on servers outside of the UK, this is the case for Australia too.

Under the UK’s privacy laws, transferring personal data to a nation without an adequate data protection framework is a breach, Australia handles cross-border information sharing much the same way. This means the consequences for staff inputting sensitive and detailed private information from clients may have profoundly serious consequences here as well.

Regardless of whether the notes were anonymous when pasted in, if consistently used, private information is being transferred overseas where detailed ‘pictures’ of clients are being accrued by those outside of a care organisations control.

Australian legislation

Australia’s Office of the Australian Information Commissioner provides stark provisions on what is and is not permitted with cross-boarder disclosure of personal information.

Within Australia’s Privacy Principles (APP), and further legislation, the guardrails hold clear responsibilities for organisations to follow, and penalties if they do not.

An APP entity is an Australian government agency or organisation, including businesses, non-profits and individuals, of which all aged care operators in Australia fall within.

Experts in law, Sprint Law, highlight that any businesses, regardless of cashflow amount, if an organisation provides health services or handles health information they are seen as an APP entity under Australian law. And subsequently, “if you’re an APP entity and you mishandle personal information, you could face investigations by the Office of the Australian Information Commissioner (OAIC), serious reputational damage, and significant penalties”.

Australian law stipulates:

“APP 8 and s 16C create a framework for the cross-border disclosure of personal information. The framework generally requires an APP entity to ensure that an overseas recipient will handle an individual’s personal information in accordance with the APPs, and makes the APP entity accountable if the overseas recipient mishandles the information. [1] This reflects a central object of the Privacy Act, to facilitate the free flow of information across national borders while ensuring that the privacy of individuals is respected (s 2A(f)).”

The problem compounds, Tinsley indicates, in that many AI developers are utlising user inputs to train and progress further iterations of their internal mechanics, what is known as their “models”. This means that client details, even though indirectly input into AI software, will likely become entrenched in a commercial AI system, blatantly breaching Australia’s data privacy laws under APP entity responsibility and data handling.

Under Australian Privacy Principle (APP) 11.2, organisations are required to take reasonable steps to eradicate or de-identify personal information if they no longer need it for any authorised purpose. If a client moves on from an organisation and requests their information be deleted by a care provider, if sustained use of AI has been maintained, an organisation may not be unable to comply, with the potential for penalty risk.

A growing risk

Tinsley echoes warnings from multiple global sectors, as the AI boom shows no signs of slowing. He details that as AI models become increasingly capable, the possibility for systems to infer identity from disjointed or fractured information is only likely to rise.

What would be difficult to connect to one senior and family today, medications, moods and preferences being unidentifiable, may very well be collated and clear in the months to come. And this is the speed that AI advancement is progressing at.

Tinsley is adamant, employers cannot assume the risk will remain static.

Management approach

In response to this growing reality of use, risk and consequences, Tinsley advocates for governance over ban. He endorses for providers to consider governance policy on AI as a worthwhile, albeit sizable, leadership consideration of the immediate. 

He details that three elements must be in place to be guardrails of AI use within care organisations.

1. Policy – he stipulates that clarity must be at the forefront for staff understanding: “A clear, written AI policy specifying which tools are approved for use with service user data, and which are not. “Don’t use ChatGPT for care notes” needs to be explicit, not implied.”

2. Training must be managed in proximity and discussion: “Staff need to understand why the policy exists, not just that it does. A worker who understands the GDPR [APP entity responsibility] risk is far less likely to cut corners than one who has simply been told no.”

3. Oversight must be embedded in operational frequency: “Spot checks, audit trails, and a culture where workers feel comfortable raising concerns. Policy without oversight is just a document. Training without oversight drifts.”

Tinsley is frank. If organisations are in the business of care, AI governance is no longer tomorrow’s problem. In the immediate, it is a core part of an organisation and leadership teams’ duty of care.