Data Labeling for Search Relevance Evaluation with Dmitry Ustalov
Dr. Dmitry Ustalov, Head of Research at Toloka shared his experience in building human-in-the-loop pipelines for information retrieval evaluation based on our full-day tutorial presented at the WWW’21 conference. There was an introduction of the ranking problem, a discussion on the commonly used ranking quality metrics, and then focus on a human-in-the-loop-based approach to obtain relevance judgments at scale. These judgments can be further used to improve the performance of search and recommender systems. Finally, he shared and discussed best practices and pitfalls from his own experience.
👉 Speaker Bio: Dr. Dmitry Ustalov is the Head of Research at Toloka, a global data labeling platform. He is responsible for enabling the state-of-the-art methods for quality control in Toloka and spreading the innovations made by the Toloka Research team.
=========================
MLT (Machine Learning Tokyo)
site: https://machinelearningtokyo
3 views
32
6
5 years ago 00:25:05 5
Data Labeling as Religious Experience
1 year ago 00:06:02 1
Подкаст TAdviser: Data labeling
6 years ago 00:04:19 4
GraphGrail | Data labeling bot MVP
4 years ago 00:06:06 4
How to Perform Data Labeling for Camera and Lidar Sensor Data
3 years ago 00:35:07 3
Data Labeling for Search Relevance Evaluation with Dmitry Ustalov
4 years ago 00:41:20 8
Learning with Limited Labeled Data | Shioulin Sam, PhD - ODSC Europe 2019
2 years ago 02:05:32 4
Topic Modeling with BERT and Automatic Cluster Labeling | NLP | Data Science | Machine Learning
4 years ago 00:14:51 13
Negative Data Augmentation
5 years ago 00:03:53 45
Waveshaper - Data Source
4 years ago 02:05:31 18
Data Viz - Matplotlib for Data Science and Machine Learning
4 years ago 00:45:56 10
Data Viz - Matplotlib for Data Science and Machine Learning (Part 2)
3 years ago 00:03:03 31
Enter Data - Midnight Blue
3 years ago 00:19:27 7
Teach Computers to Connect Videos and Text without Labeled Data - VideoClip
5 years ago 00:03:51 39
Big Data - Unglued
4 years ago 00:18:36 34
QGIS Import XYZ coordinates data, Create DEM, Create contours, Add Contours label, Use 3D view model
3 years ago 00:09:40 7
45 - Label Smoothing in Deep Learning | Machine Learning | Statistics | Data Science
6 years ago 02:04:54 9
Lesson 3: Deep Learning 2019 - Data blocks; Multi-label classification; Segmentation
1 year ago 00:04:10 2
Create MEANINGFUL data labels with measures in Power BI
4 years ago 00:59:03 47
IDAO 2020 Final, “Data labelling on a large scale Must-have expertise of modern AI specialist”
10 months ago 00:02:34 2
Create Custom Data Labels In Power Bi For More Control Over Your Data Visualization!