Deepfake arrest lends weight to stronger GenAI safeguards, says eSafety Commissioner

Following the arrest of a Victorian grammar school student for the creation and distribution of classmate deepfakes, eSafety commissioner Julie Inman Grant has argued that the creators of AI tools need to do more.

Editor’s note: This story first appeared on HR Leader’s sister brand, Cyber Daily.

Australia’s eSafety commissioner, Julie Inman Grant, has called on “the purveyors and profiteers of AI” to shoulder a greater share of the burden when it comes to protecting individuals from deepfake abuse.

Inman Grant made the comments following the arrest of a Victorian schoolboy for creating and distributing deepfake images of about 50 female students at Bacchus Marsh Grammar.

Police learnt of the incident on 7 June and – after arresting and releasing the boy – said the investigation was “ongoing”. A mother of a girl who had seen the images – but was not in them – told The Guardian that her first reaction to the incident was “to be sick”.

“I mean they are children … the photos were mutilated, and so graphic,” the woman said. “I almost threw up when I saw it.”

Inman Grant said that it was far from the first case of “deepfake image-based abuse” reported to her office.

“We are already receiving reports containing synthetic (AI-generated) child sexual abuse material, as well as deepfake images and videos created by teens to bully their peers and, of course, ‘deepfaked porn’ through our image-based abuse scheme, which we are referring to as ‘deepfake image-based abuse’,” Inman Grant said in a statement.

“We can provide real help to Australians who fall victim to image-based abuse, including deepfakes, and we have a very high success rate in getting this distressing material down,” Inman Grant added, before calling on the AI industry to focus on “safety by design”.

“But a greater burden must fall on the purveyors and profiteers of AI to take a robust approach so they are engineering out misuse at the front end – something we call safety by design.”

“We are not going to regulate or litigate our way out of this – the primary digital safeguards must be embedded at the design phase and throughout the model development and deployment process.”

Jacinta Allan, the Premier of Victoria, said her thoughts were with the girls and their parents.

“Women and girls deserve respect in class, online and everywhere else in our community, which is why we have made laws against this behaviour, and we are teaching respectful relationships in schools to stop violence before it starts,” Allan said.

The school’s acting principal, Kevin Richardson, said in a statement that “Bacchus Marsh Grammar is taking this matter very seriously and has contacted Victoria Police”.

“The wellbeing of Bacchus Marsh Grammar students and their families is of paramount importance to the school and is being addressed. All students affected are being offered support from our wellbeing staff.”