User Testing at Deutsche Welle

By on October 29th, 2020 in Blog Posts

Overview

WeVerify testing ran from December 2019 and will continue until September 2021. In total, nine test cycles have been scheduled in order to evaluate the tools and single components from a user, i.e. fact checker’s, perspective.

In these test cycles, single components (e.g. Twitter Social Network Analysis Tool) and tools such as the browser plug-in that combine several components have been tested. Staff from the user partners of the project (AFP, EU Disinfo Lab and Deutsche Welle) that verify user-generated content on a daily basis have validated this technology to provide feedback on its usefulness, and also suggest improvements.

As WeVerify approaches the end of its second project year, four test cycles have been completed so far. This article outlines the results focused on three major developments of the project: the InVID/WeVerify Browser plug-in, the Database of Known Fakes, and the Professional Workbench.

Five more test cycles are scheduled for the final year.

InVID/WeVerify Browser Plug-in

The initial version of the browser plug-in was developed during the EU-funded project InVID (invid-project.eu). The tool is designed to be a verification “Swiss army knife,” helping journalists save time and be more efficient in their fact-checking tasks.

Figure 1 Overview of Browser Plug-in Features

Over the course of WeVerify testing and development, the browser plug-in has been enhanced with more modules designed to facilitate the fact checker’s work. The Twitter Social Network Analysis Tool and the Digital Companion are examples of two such additions.

The Twitter Social Network Analysis Tool is an attempt to democratize social network analysis on Twitter to empower journalists and fact-checkers in their investigations and debunking tasks while tackling disinformation. By reducing complexity and providing automated visualizations that can be embedded into a fact-checking report, users can quickly make sense of content propagation on Twitter and gradually improve their understanding of social network analysis.

The Digital Companion is designed to allow non-professional users, such as private persons with an interest in verifying content on social media platforms, to undertake first steps to identify content manipulation. When using this tool, the user is guided through initial verification steps, helping them critically question whether a piece of content is what it claims to be.

During the final project year, WeVerify will go one step further by implementing the open-source micro-targeted Debunking Bot. This will help users who have encountered misinformation explore debunks, fact checks and credible content.

Database of Known Fakes (DBKF)

The blockchain-based Database of Known Fakes aims to create and maintain a consistent, distributed database of manipulated content. The web-based user interface enables journalists and verification professionals to add newly identified fake content and search for already known fakes. The DBKF is currently in its beta version and will be made publicly available over the next few months.

Figure 2 Screenshot of DBKF Beta Version

Professional Workbench

The Professional Workbench builds upon the commercial tool Truly which provides journalists with a working environment where they can collaboratively verify online content. It is a web-based collaboration and verification platform.

Figure 3 Screenshot of Professional Workbench

The platform offers easy aggregation of user-generated content, real time collaboration, a clear verification workflow, integrated semi-automated verification tools and shared verification tasks and information between teams. This collaboration can occur both within the same organization (such as news desks) or even across departmental and organizational boundaries (such as between multiple media organizations). The platform is continuously updated with new features, such as deepfake detection, to tackle new challenges encountered by professionals working in the field.

Feedback from Users

Feedback from those who have tested the tools and modules shows that the project is well on track. In general, users have reported appreciation for the tools’ ease of use, specifying that they require little-to-no technical knowledge and make certain aspects of analyzing social media data clear and visual.

Staff from Deutsche Welle’s social media desk see a need for increased understanding of how misinformation is disseminated on Twitter and other social media channels.

One of the most important elements of WeVerify, according to users, is that features are increasingly interlinked, making the browser plug-in a ‘one-stop-shop’ for much of their work. The Twitter Social Analysis Tool has been named a real standout with the search giving a quick overview of what is happening in specific circles of Twitter while also providing a variety of ways to analyze this information. The ability to export interaction graphs to conduct a deeper network analysis is very useful for journalists who want more technical freedom.

Users report that the Database of Known Fakes saves time and effort by creating an archive of false claims, preventing duplicate fact-checking work. The Professional Workbench helps journalists work together on more complex fact-checks.

Users have also identified a few drawbacks, which are mostly due to part of the technology not being fully operational yet. Reporting of bugs and shortcomings has provided the developers with information needed to further improve the technical solutions.

One wish users had is the ability to search links to see where they have been shared online. Alongside integration with the other tools, this addition would ensure users spend much more time with WeVerify when conducting their initial research.

The key highlight mentioned by testers is the intuitive GUI, which is key for journalists who are not tech-savvy.

In the final project year, the frequency of test cycles will be increased. More external users from other media outlets and human rights organizations will be included in test sessions to improve the quality of WeVerify’s tools and solutions.

Authors: Jordan Wildon (DW) and Tim Koch (DW)

Photo by You X Ventures on Unsplash

  • Share:

Leave a Comment

sing in to post your comment or sign-up if you dont have any account.

WeVerify.eu Privacy Policy

1. Purpose

The purpose of this Privacy Policy is to describe what we collect, use and sometimes share information about you through our online interfaces (e.g., websites and email) owned and controlled by us, including WeVerify and all subdomains (collectively referred to herein as the “Site”).

At WeVerify, we believe that you should have control of your data. Control starts with information. This is why you should know what data we collect from you and how we use it.

This notice and the accompanying policy is to enable you to make the best decisions about the information that you choose to share with us.2

2. Privacy Policy

By accessing and using any of WeVerify site, demonstrators or publicly available services, you expressly and knowingly consent to the information collection and use practices as described in this Privacy Policy.

3. Our Privacy Commitment

Our commitment to your privacy, is based on the following principles which we apply to our use of both your personally identifiable data (“Your Personal Data” or generally “Personal Data”) and to certain anonymous information we collect when you visit our Sites (“Technical Information”, and together with Personal Data, “Your Data”):

  • We will describe Your Data we will collect;
  • We will inform you clearly about our collection and use of Your Data;
  • We will either seek your express informed consent or rely on other legally permissible bases for the use of Your Data – either way, we will inform you of the basis for our use of Your Data;
  • We will give you control over the privacy preferences that apply to Your Data, including the rights to (a) change your mind about our use, (b) have access to change or correct inaccurate aspects of Your Data, and (c) require that we delete all or parts of Your Data (d) request Your Data in a portable format;
  • We will not sell or rent Your Personal Data to others;
  • We endeavor to maximize the protection of Your Data, and provide you with prompt notice in the unlikely event that a data loss incident or breach occurs; and
  • We will endeavor to be completely transparent and open about our data privacy policies and practices.

4. What Information does this Privacy Policy cover?

This Privacy Policy covers information we collect from you through all of our channels, including website, email and others. Some of our website’s functionality can be used without revealing any of Your Data. In order to access certain products, demonstrators or services, you may need to submit, or we may collect information that can be used to identify you.

Your Personal Data can include information such as your name and email address, among other things. You are responsible for ensuring the accuracy of the information you submit to us. Inaccurate information may affect your ability to use the site, download products, any follow-up information you request, and our ability to contact you. For example, your email address should be kept current because that is one of the primary manners in which we communicate with you.

5. How do we collect information?

We collect Your Data in the following ways:

  • You give it to us when you download software or documentation; register for an event such as a webinar; sign up for a newsletter; fill a form on the website or via any other sales or marketing channel;
  • You give it to us by email or phone inquiries or
  • We automatically collect Technical Information when you visit our Sites.

6. What information do we collect?  

When filling any form on the website, we collect Personal Data such as name; phone; email; company name, website and address; job title and category; social media data and nature of the interest.

In addition, we enrich the Personal Data above with Technical Information, related to:

  1. Conversion point (when, where, what campaign, source);
  2. Activity (dates of contact, email opens, link clicks, website visits, etc.);
  3. Opt-in Date to trace your consent;
  4. CRM identifiers;

When processing payments we additionally collect VAT ID, company identification and other information for invoicing and tax purposes.

7. How do we use Your Data collected at our sites?

We will use Your Data to:

  • Provide information, product or a service requested or consented to by you.
  • Comply with relevant contractual obligations with you and other third parties.
  • Improve Site performance and content, including troubleshooting and diagnostics.
  • Improve our engagement and interaction with you.
  • Facilitate your attendance at and participation in our events, communities or blogs.
  • Process a request or payment submitted to us.
  • Comply with legal requests.

8. What are your rights to control Your Data?

You have the right to request that we:

  • provide access to any of Your Personal Data we hold about you;
  • prevent the processing of Your Personal Data for direct marketing purposes;
  • update any of Your Personal Data which is out of date or incorrect;
  • delete Your Personal Data which we are holding about you;
  • restrict the way that we process Your Personal Data;
  • provide Your Personal Data to a third party provider of services; or
  • provide you with a copy of Your Personal Data which we hold about you.

We try to answer every email promptly where possible and provide our response within the time period stated by applicable law. Keep in mind, however, that there will be residual information that will remain within our databases, access logs and other records, which may or may not contain Your Personal Data. Please also note that certain parts of Your Personal Data may be exempt from such requests in certain circumstances, which may include if we need to keep processing Your Personal Data to comply with a legal obligation.

When you email us with a request, we may ask that you provide us with information necessary to confirm your identity.

8. What data do we retain?

We will only retain Your Data stored on our servers in accordance with the legitimate needs of our business and as required or permitted by applicable law. We will not retain any unused Personal Data on our systems longer than necessary for legitimate business purposes.