AI TOOLS IMPLEMENTATION IN CONTEMPORARY INTERPRETATION
DOI:
https://doi.org/10.32782/2522-4077-2025-213-54Keywords:
interpretation, machine translation, information technology, artificial intelligence, AI dubbing, manipulationsAbstract
The paper has been devoted to the study of the newest, hybrid type of interpretation with the use of technical means, namely, machine dubbing, a striking case of which public could see in the resonant interview of the President of Ukraine Volodymyr Zelenskyy given to the well-known American podcaster Lex Fridman in December 2024. A complex algorithm for providing interpretation was used in the process of the interviewing. It involved a number of intermediate stages that were not totally clear to its viewers and listeners, in particular, the production of generated voices which were trained and made on the basis of real ones. The subsequent dissemination of the interview materials (multilingual AI-dubbed videos with proper subtitling) made a mixed impression even on the professionally trained audience who was unable to establish the mechanism of how public materials were created. Such cases could potentially open some dangerous opportunities for the informational, social and political manipulation by means of audiovisual products. Moreover, such practice is fully authorised by the respective projects participants on both sides. It seems to be obviously a consequence of their lack of understanding how dangerous these documentary AI-enhanced interviews could be for the society. Machine translation errors, compounded by the process of creating multilingual transcripts and their subsequent dubbing (voiceover) with the help of artificially generated voices, cannot be distinguished from real ones without special training and equipment. It can be possibly reduced to the everyday practice of creating news or journalistic materials. Unlike illegal deepfakes, whose IT nature is the same, machine translations and AI dubbing of documentary interviews are allowed to be published by their participants, which opens significant opportunities for covert manipulation of public opinion and public assessment of such materials capable of spreading harmful narratives as part of defamation and propaganda campaigns.
References
Семенова І. «Такий він, всесвіт Трампа». До кого насправді звертався Зеленський у подкасті Фрідмана і як реагують в Україні на різке інтерв’ю. NV, 06.01.2025. URL: https://nv.ua/ukr/ukraine/politics/interv-yu-zelenskogo-fridmanu-dlya-kogo-zroblene-i-yaka-reakciya-ukrajinciv-novini-ukrajini-50479397. html (дата звернення: 24.04.2025).
Fridman L. «Dear President @ZelenskyyUa, I look forward to our 3 hour conversation» [Твіт]. Twitter, 24.12.2024. URL: https://x.com/lexfridman/status/1871502609451778181 (дата звернення: 24.04.2025).
The 2022 Nimdzi Language Technology Atlas. Nimdzi. URL: https://www.nimdzi.com/nimdzi-language-technology-atlas-2022/ (дата звернення: 24.04.2025).
Matoušek J., Hanzlíček Z., Tihelka D., Méner M. Automatic dubbing of TV programmes for the hearing impaired. IEEE 10th International Conference on Signal Processing Proceedings, Beijing, China. 2010. P. 589–592. https://doi.org/10.1109/ICOSP.2010.5655861.
Matoušek J., Vít J. Improving automatic dubbing with subtitle timing optimisation using video cut detection. 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Kyoto, Japan. 2012. P. 2385–2388. https://doi.org/10.1109/ICASSP.2012.6288395.
Garrido P., Valgaerts L. et al. VDub: Modifying Face Video of Actors for Plausible Visual Alignment to a Dubbed Audio Track. Computer Graphics Forum. 2015. Vol. 34, Issue 2. May. P. 193–204. https://doi.org/10.1111/cgf.12552.
Chung J. S., Jamaludin A., Zisserman A. You said that?, 2017. Ithaca, NY : Cornell University arXiv. 12 p. (Препринт 10.48550/arXiv.1705.02966). https://doi.org/10.48550/arXiv.1705.02966
Deepdub. «Experience Localization with Real Emotions» [допис]. LinkedIn, 2024. URL: https://www.linkedin.com/posts/deepdub-ai_vanda-the-first-ai-dubbed-drama-by-legendary-activity-7186270478662012928-livX/ (дата звернення: 24.04.2025).
Holliday C. Ghosts in the Celluloid: AI Video Dubbing and TrueSync. JCMS: Journal of Cinema and Media Studies. 2024. Volume 64, Issue 1, Fall. P. 175–182. https://doi.org/10.1353/cj.2024.a944431.
Kumar K. S., Aravindhan S., Pavankumar K., Veeramuthuselvan T. Autodubs: Translating and Dubbing Videos. Emerging Trends in Expert Applications and Security. ICETEAS 2023. Lecture Notes in Networks and Systems / ed. by Rathore V. S., Piuri V., Babo R., Ferreira M. C. Vol. 682. Singapore, 2023. P. 53–60. https://doi.org/10.1007/978-981-99-1946-8_6.
Chenxu H., Qiao T., Tingle L., Yuping W., Yuxuan W., Hang Z. Neural Dubber: Dubbing for Videos According to Scripts, 2021. Ithaca, NY : Cornell University arXiv. 14 p. (Препринт 2110.08243). https://doi.org/10.48550/arXiv.2110.08243
Hsu J. AI Modifies Actor Performances for Flawless Dubbing Commercialization of AI-assisted dubbing for TV and films hints at bigger possibilities with digitally captured acting performances. IEEE Spectrum. URL: https://spectrum.ieee.org/ai-modifies-actor-performances-for-flawless-dubbing (дата звернення: 24.04.2025).
Gerken T. Gaming voice actors blindsided by ‘garbage’ union AI deal. BBC. URL: https://www.bbc.com/news/technology-67922303 (дата звернення: 24.04.2025).
Markowitz D. AI Dubs Over Subs? Translating and Dubbing Videos with AI. Dale on AI. URL: https://daleonai.com/translate-dub-videos-with-ml (дата звернення: 24.04.2025).
Adee S. What Are Deepfakes and How Are They Created? Deepfake technologies: What they are, what they do, and how they’re made. IEEE Spectrum. URL: https://spectrum.ieee.org/what-is-deepfake (дата звернення: 24.04.2025).
Amato G., Behrmann M. et al. AI in the media and creative industries, 2019. Ithaca, NY : Cornell University arXiv. 34 p. (Препринт 1905.04175). https://doi.org/10.48550/arXiv.1905.04175
Apolo Y., Michael K. Beyond A Reasonable Doubt? Audiovisual Evidence, AI Manipulation, Deepfakes, and the Law. IEEE Transactions on Technology and Society. 2024. Vol. 5, No. 2, June 2024. P. 156–168. https://doi.org/10.1109/TTS.2024.3427816.
Mihalache I. Ethical Analysis of AI-Enhanced Audiovisual Translation Tools. 2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC), Osaka, Japan, 2024. P. 1558–1559. https://doi.org/10.1109/COMPSAC61105.2024.00237.
This Voice Doesn’t Exist – Generative Voice AI. ElevenLabs, 11.01.2023. URL: https://elevenlabs.io/blog/enter-the-new-year-with-a-bang (дата звернення: 24.04.2025).
Slodov A. «Milei’s 2024 Davos talk, directly translated to English by AI» [Твіт]. Twitter, 18.01.2024. URL: https://x.com/aphysicist/status/1747868626948907325 (дата звернення: 24.04.2025).
Agranovich A., Nachmani E., Rybakov O., Ding Y., Jia Y., Bar N., Zen H., Ramanovich M. T. SimulTron: On-device simultaneous speech to speech translation, 2024. Ithaca, NY : Cornell University arXiv. 5 p. (Пре-принт 2406.02133). https://doi.org/10.48550/arXiv.2406.02133
Fuster J. For Voice Actors, the Race Against AI Has Already Begun. The Wrap. URL: https://www.thewrap.com/ai-dubbing-voice-actors/ (дата звернення: 24.04.2025).
Cinematic breakthrough: “THREE” Makes History as First Arabic Film Dubbed with AI. Camb.ai URL: https://www.camb.ai/blog-post/cinematic-breakthrough-three-makes-history-as-first-arabic-film-dubbed-with-ai (дата звернення: 24.04.2025).
Titan B. The world’s first movie fully translated and dubbed by AI is about to be released. Medium, 20.07.2024. URL: https://braintitan.medium.com/the-worlds-first-movie-fully-translated-and-dubbed-by-ai-is-about-to-be-released-a29e88a5b298 (дата звернення: 24.04.2025).
Language Technology Radar. Curated catalogue of language technology providers. Nimdzi. URL: https://www.nimdzi.com/language-technology-radar/?preview=true&_ltr_audiovisual_translation=ai-enhanced-dubbing-tools (дата звернення: 24.04.2025).
Інтерв’ю Володимира Зеленського американському подкастеру Лексу Фрідману [Відео]. Офіс Президента України. YouTube, 05.01.2025. URL: https://www.youtube.com/watch?v=8XEpoCMIADI (дата звернення: 24.04.2025).
#456 – Volodymyr Zelenskyy: Ukraine, War, Peace, Putin, Trump, NATO, and Freedom. Lex Fridman. URL: https://lexfridman.com/volodymyr-zelenskyy/ (дата звернення: 24.04.2025).
Fridman L. «Hi everyone, I would like to fix a translation in the audio / subtitle that better captures what President Zelenskyy was saying». Telegram, 06.01.2025. URL: https://t.me/lexfridman/329 (дата звернення: 24.04.2025).
«Хтось дивився інтервю Президента України Лексу Фрідману?» [оновлення статусу]. Українські перекладачі. Facebook, 12.01.2025. URL: https://shorturl.at/QuW0C (дата звернення: 24.04.2025).
Мазур Ю. «Враховуючи, скільки людей спитало (та скільки, певно, промовчало), мушу офіційно заявити: ні, Фрідмана я не перекладав» [оновлення статусу]. Facebook, 07.01.2025. URL: https://shorturl.at/wq834 (дата звернення: 24.04.2025).
Казанцев Д. ШІ-сервіс ElevenLabs переклав інтерв’ю Зеленського Лексу Фрідману. Що це за стартап та як він працює? Межа, 09.01.2025. URL: https://mezha.media/articles/shi-servis-elevenlabs-iak-vin-pratsiuie/ (дата звернення: 24.04.2025).
Jimenez, J. AI company promises changes after ‘voice cloning’ tool used to make celebrities say awful things. PC Gamer, 01.02.2023. URL: https://www.pcgamer.com/ai-company-promises-changes-after-voice-cloning-tool-used-to-make-celebrities-say-awful-things/ (дата звернення: 24.04.2025).
Deepdub. Deepdub unveils first-ever AI dubbing technology with precise control of characters’ accents. PR Newswire. URL: https://www.prnewswire.com/news-releases/deepdub-unveils-first-ever-ai-dubbing-technology-with-precise-control-of-characters-accents-302075611.html (дата звернення: 24.04.2025).
Шаховал З. Використання штучного інтелекту у підготовці майбутніх перекладачів. Матеріали XIV Міжнародної науково-практичної конференції «Actual problems of personality psychology in the modern world», Рим, Італія. 2024. С. 263–267.
