{"id":20,"date":"2022-04-29T14:32:54","date_gmt":"2022-04-29T11:32:54","guid":{"rendered":"https:\/\/traces.gate-ai.eu\/?page_id=20"},"modified":"2023-03-14T13:40:47","modified_gmt":"2023-03-14T10:40:47","slug":"results","status":"publish","type":"page","link":"https:\/\/traces.gate-ai.eu\/?page_id=20","title":{"rendered":"Results"},"content":{"rendered":"\n

Videos<\/h2>\n\n\n\n

You can watch a short video about the project’s results on this link<\/a>.<\/p>\n\n\n\n

\"\"<\/a><\/figure>\n\n\n\n

<\/p>\n\n\n\n

<\/p>\n\n\n\n

Models & Tools<\/h2>\n\n\n\n

We implemented and trained three machine learning models. One that can detect Bulgarian texts, generated by the GPT-2 and ChatGPT language models, one for detecting disinformation, and one for detecting untrue information. All three models are trained with data in Bulgarian language.<\/p>\n\n\n\n

We also created a tool that uses the three models. The tool has three versions. All three versions use the same model for disinformation detection, but different models for textual deepfakes detection, and the last version contains a model for detecting untrue information.<\/p>\n\n\n\n

Current release:<\/em><\/p>\n\n\n\n

TRACES-tool v1.2<\/em> :<\/strong> The deepfake model can detect texts generated with the models GPT-2 and ChatGPT, disinformation and untrue information. If a text is recognized as both generated by the language models and containing untrue information or disinformation, it can be considered as potentially being a textual deepfake. The model for detecting untrue information has F1-Score of 0.96.<\/p>\n\n\n\n

Previous releases:<\/em><\/p>\n\n\n\n

TRACES-tool v1.1<\/em> :<\/strong> The deepfake model can detect texts generated with the models GPT-2 and ChatGPT, and disinformation. We used 10-fold cross-validation and achieved F1-Score of 0.88. <\/p>\n\n\n\n

TRACES-tool<\/em><\/strong> v1.0 :<\/em><\/strong> The deepfake model can detect texts, generated by GPT-2 with F1-Score 0.9339.<\/p>\n\n\n\n

To test the tool, please follow the link to the DEMO<\/strong>.<\/a> See the screenshot of the tool, containing the input text in Bulgarian with the following translation into English: “I have no idea where did the cookies disappear”.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Note:<\/strong> \u0422he code of the tool contains an unpublished third party model for detecting disinformation that will be released it in the next few months.<\/em> Because of this reason, the tool cannot be yet publicly released, but it is available for testing.<\/em><\/p>\n\n\n\n

<\/p>\n\n\n\n

We have publicly released two machine learning models (the ML models with the highest F1-Scores): <\/strong><\/p>\n\n\n\n

    \n
  1. Machine learning model for identifying Bulgarian texts, automatically generated by the language models GPT-2 and ChatGPT, achieving F1-Score 0.88. Can be accessed under legal restrictions on Zenodo<\/a>.<\/li>\n\n\n\n
  2. Machine learning model for identifying Bulgarian texts, potentially containing untrue information, achieving F1-Score 0.96. The model can be accessed under strict legal conditions on Zenodo<\/a>.<\/li>\n<\/ol>\n\n\n\n

    <\/p>\n\n\n\n

    <\/p>\n\n\n\n

    Datasets<\/h2>\n\n\n\n

    Bulgarian Twitter dataset on Covid-19, annotated with linguistic markers of lies<\/a><\/p>\n\n\n\n

    Bulgarian Telegram dataset, annotated with linguistic markers of lies<\/a><\/p>\n\n\n\n

    Bulgarian Twitter dataset on lies and manipulation, annotated with linguistic markers of lies<\/a><\/p>\n\n\n\n

    Bulgarian Twitter Dataset on Famous Bulgarian Political Cases of Suspected Lies, Annotated with Linguistic Markers of Lies<\/a><\/p>\n\n\n\n

    Bulgarian sentiment analysis Twitter dataset<\/a><\/p>\n\n\n\n

    Python Scripts for Downloading from Twitter, Cleaning, and Annotating with the Linguistic Markers of Deception New (Bulgarian) Text Datasets<\/a><\/p>\n\n\n\n

    Hierarchical Classification of Categories of Linguistic and Psycholinguistic Markers of Deception with Bulgarian Expression Lists for Disinformation Detection<\/a><\/p>\n\n\n\n

    TRACES Telegram and Twitter Dataset with Bulgarian Journalists Manual Annotations of True\/Untrue and Disinformation\/Not and Automatic Annotations for Markers of Lies<\/a><\/p>\n\n\n\n

    <\/p>\n\n\n\n