Officials at the Daegu branch of the National Election Commission conduct monitoring activities to identify possible violations of the Public Official Election Act in online content, Feb. 8. Yonhap

Artificial intelligence (AI) generated deepfakes represent a substantial threat to political campaigns in the lead-up to the April 10 parliamentary elections, prompting concerns among experts regarding their potential to influence election outcomes.

The National Election Commission (NEC) said on Monday that it had identified 129 instances of election-related content utilizing deepfake technology from Jan. 29 to Feb. 16. Each of these cases was determined to be in violation of the Public Official Election Act.

Deepfakes, a term derived from "deep learning" and "fake," refers to images, audio and videos that closely mimic reality through the utilization of AI.

Although AI-generated synthetic content has proven to be useful in various fields, its exploitation in politics, notably for the dissemination of misinformation, presents a significant challenge for government authorities and policymakers.

"The most concerning scenario would involve uploading a deepfake video defaming a candidate just a day before the election, and it gathers millions of views. There would be no time for the press or the government to verify it before voters head to the polls the next day," said Jeon Chang-bae, chairman of the International Artificial Intelligence Ethics Association (IAIE), a Seoul-based non-profit group studying AI ethics.

Kim Myung-joo, a professor of information security at Seoul Women's University, said, "AI-generated misinformation could have a significant impact on swing voters' perceptions, among others."

The professor also pointed out that crafting a convincing hoax using deepfake technology has become remarkably easy.

"Creating deepfake videos is a bit complicated, but anyone can generate a deepfake image using a mobile app. It only takes about 10 minutes. This means that political YouTubers or bloggers can easily use this technology to spread misinformation about politicians they dislike," he said.

A screenshot of a deepfake clip of President Yoon Suk Yeol endorsing a candidate for the head of Namhae County in the lead-up to the June 1 local elections in 2022 / Korea Times file

Deepfake content is not entirely new to Korean politics.

During the presidential election campaign in 2022, then candidates Yoon Suk Yeol and Lee Jae-myung utilized AI-generated avatars of themselves to announce their pledges and appeal to young voters.

However, there have been cases of the misuse of deepfake content in politics.

During the local elections in May 2022, a manipulated video clip depicting Yoon endorsing a conservative Namhae County head candidate circulated on social media. This led viewers to mistakenly believe that the president had failed to remain politically neutral.

More recently, a deepfake video uploaded on YouTube depicted the interim leader of the ruling People Power Party (PPP), Han Dong-hoon, likening liberals to gangsters during a press conference.

In response to these incidents, a special parliamentary committee passed a revision to the Public Official Election Act in December, aimed at prohibiting political campaign content that utilizes AI-generated deepfakes during the election season. The law came into effect on Jan. 29.

Under the revised law, individuals involved in distributing or showcasing political campaign videos crafted using deepfake technology within the 90-day window leading up to an election could potentially incur severe penalties. This includes a maximum prison term of seven years or a fine reaching up to 50 million won ($37,420).

While it is encouraging that bipartisan support has led to a legal ban on the use of deepfakes in election campaigns in Korea – in contrast to the United States or European states that primarily rely on self-regulation by big tech companies – the existing laws are insufficient to prevent generation of deepfakes in the first place, experts said.

"It's a big step forward. However, what is more important than the laws is raising awareness among the public that any videos and images they view on the internet could be fabricated, and not trusting them at face value," Jeon said, stressing the importance of digital media literacy and critical thinking.