YouTube Block Utility Rating Guidelines

Version 1.2 January 10, 2013 Introduction ..................................................................................................................... 1 Important Notes ................................................................................................................ 2 Using the Utility Slider and Assigning Flags ............................................................................ 2 Types of YouTube Search Results ........................................................................................ 3 Video Results ................................................................................................................... 6 Channel Results ............................................................................................................... 7 Playlist Results ............................................................................................................... 10 Movie Results ................................................................................................................ 12 Pay-Restricted Results ..................................................................................................... 13 Rating Overview ............................................................................................................. 15 Evaluating Video Result Blocks ......................................................................................... 16 Video Result Block Rating Examples ................................................................................... 19 Evaluating Channel Result Blocks ...................................................................................... 20 Channel Result Block Rating Examples ............................................................................... 21 Evaluating Playlist Result Blocks ....................................................................................... 23 Playlist Result Block Rating Examples ................................................................................ 25 Evaluating Movie Result Blocks ......................................................................................... 26 Movie Result Block Rating Examples .................................................................................. 26 Evaluating Pay-Restricted Results ...................................................................................... 27 Pay-Restricted Result Block Rating Examples ....................................................................... 27 Assigning Flags to YouTube Result Blocks .......................................................................... 28 Reporting Results with Duplicate Landing Pages ................................................................... 32 Assigning an Overall Side by Side Score (if requested) ........................................................... 33

1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23.

1.

Introduction

The goal of YouTube search is to provide users with relevant, engaging videos. The data derived from your rating tasks is used to inform YouTube search quality engineers about what can be done to achieve this goal. These guidelines are designed to help make your responses on rating tasks as useful as possible. In YouTube Block Utility tasks, you will be given a query that was issued on www.youtube.com and one or two sets of YouTube result blocks to evaluate. In all YouTube Block Utility tasks, you will: u=imkhhj 1. Assign a Utility Rating to each YouTube result block using the given Utility Rating slider. 2. Assign Flags to YouTube result blocks, when appropriate. 3. Report Duplicate Result Blocks (results that have the same landing page). In some YouTube Block Utility tasks, you will also be asked to assign an Overall Side by Side rating. You will be required to leave a comment if you think one side is better than the other side. These guidelines provide information about the different types of results provided by YouTube and how to evaluate them.

Proprietary and Confidential - 2013

2.

Important Notes

Result blocks are sometimes slow to load: Before assigning any ratings, please scan the task page to make sure that all result blocks have loaded. Please note that result blocks may be slow to load and/or that the task page may need reloading if there are empty result blocks. If the result blocks fail to load after refreshing a few times, please release the task and send a note to your employer. Sometimes a side will not have any result blocks: If one side displays the message This side did not generate any results, please do not release the task or send a note to your employer. Please assign utility ratings to each result block using your best judgment. Please also assign an overall SxS rating, if the task asks for one. Sometimes the task will not be divided into two sides: You will be given just one set of result blocks to evaluate. Do not refresh the task page after assigning ratings: If you refresh the page after assigning ratings, you will lose them and they will have to be entered again.

3.

Using the Utility Slider and Assigning Flags

In YouTube block utility tasks, you will rate EVERY result independently using the Utility Rating sliding scale (slider) provided. In other words, you will rate each result as if it were a separate rating task. Sliding Scale Rating Abbreviations: To make the interface more compact, we have abbreviated some of the labels. (N/A = Unratable, Useless, Marg.U = Marginally Useful, Useful, Very.U = Very Useful). You will learn more about the YouTube utility rating scale in later sections of these guidelines. This is what the YouTube Utility Rating slider looks like:

In-Between Ratings: Please note that you are allowed to assign "in-between" ratings (for example, there are 3 small tick marks in between Useless and Marg.U on the slider). Feel free to use them if you think the utility of a YouTube video result falls between two labels. You can either drag the slider or click on the point or label that you want the slider to land on. Assigning Flags: You should also assign flags (D = Doesnt play, P = Porn/offensive, S = Spam, F = Foreign Language, O = Old/Stale, A = Authoritative/Official) as needed for the results. Please do not use the F flag for results from other countries that are in the task language. For example, results from Australia in English are not foreign for English (US). This is how flags are displayed in a YouTube block utility task:

You will learn about these flags in Section 18 of these guidelines.

Proprietary and Confidential - 2013

4.

Types of YouTube Search Results

These guidelines provide guidance on

Most YouTube search results are videos, channels, playlists, or movies. rating these four types of video results.

You will be able to distinguish among these four types of video results by looking at their URLs:
Result Type Pattern to Recognize Example URLs http://www.youtube.com/watch?v=PNuctBH9uNc&fe ature=relmfu All YouTube videos have URLs that fit this pattern: Video http://www.youtube.com/watch?v= + [11-character document id] + [optional parameters] http://www.youtube.com/watch?v=0C3W7yQ8bTY&f eature=relmfu http://www.youtube.com/watch?v=W795W63n7mA&f eature=related http://www.youtube.com/user/theroyalchannel? All YouTube channels have URLs that fit this pattern: Channel http://www.youtube.com/user/ + username? http://www.youtube.com/user/whitehouse? http://www.youtube.com/user/madonna?

Topic Channel

Topic channels are a special type of channel result. (You will learn more about topic channels in Section 6.) All YouTube topic channels have URLs that fit this pattern: http://youtube.com/topic/ + [11-character document id] + [channel_name]

http://www.youtube.com/topic/wsGNZuBKZL0/chocol ate-brownie http://www.youtube.com/topic/G2a8R0cSa5M/ballet http://www.youtube.com/topic/3ksOejq4Z1A/laughing -baby http://www.youtube.com/watch?v=e3i6LSW_Kyg&list= UUYxRlFDqcWM4y7FfpiAN3KQ

Playlist

All YouTube playlists have URLs with list= in them.

http://www.youtube.com/watch?v=JGPQcGu2Cso&list= UUGm3CO6LPcN-Y7HIuyE0Rew http://www.youtube.com/watch?v=qr5VqBP1voA&list=F L81VD6eeuLLSfyY_D-N8sVw

All YouTube movie results have URLs that fit this pattern: Movie http://www.youtube.com/movie?v= + [11-character document id] + [optional parameters]

http://www.youtube.com/movie?v=wvzDHGLEUyw&feat ure=mv_sr http://www.youtube.com/movie?v=JgCtmFOEsIw&fe ature=mv_sr http://www.youtube.com/movie?v=-ssId2r7F1I

Please note that even if a video is displayed on a channel or is part of a playlist, it is still a video if it starts with: http://www.youtube.com/watch?v=.

Proprietary and Confidential - 2013

This is how YouTube result blocks for these result types were displayed at the time these guidelines were written: Result Type
Video Result Notice the HD label. This is a high definition video.

Result Block

Video Result Notice that there is no HD label on this result. This video is not a high definition video.

Channel Result Notice the CHANNEL label

Topic Channel Result Notice the CHANNEL label, as well as YouTube in the title

Playlist Result Notice the PLAYLIST label

Movie Result Notice the MOVIE label

Proprietary and Confidential - 2013

Please note: Video results may or may not show the HD label. This label merely signifies that the video is available in high definition. Channel results always show the CHANNEL label. Topic channel results always show the CHANNEL label, as well as YouTube in the title. Playlist results always show the PLAYLIST label. Movie results always show the MOVIE label.

YouTube provides other types of search results, such as show and artist results, and you may occasionally be asked to rate these other result types. Please use your judgment and assign reasonable ratings based on the utility of the landing page to the query. You will be able to distinguish these result types by their URL patterns:
Result Type Pattern to Recognize All YouTube show results have URLs that fit this pattern: Show http://www.youtube.com/show?p= + [11-character document id] + [optional parameters] All YouTube artist result have URLs that fit this pattern: Artist http://www.youtube.com/artist?a= + [11-character document id] + [channel_name] Example URLs http://www.youtube.com/show?p=IEVZPYfvjrc &tracker=show_av http://www.youtube.com/show?p=VboXbjdM36 c&tracker= http://www.youtube.com/artist?a=QrFlYXe6zII& feature=artistob http://www.youtube.com/artist?a=WzkOV8OF0j g&feature=artistob

This is how YouTube result blocks for these result types were displayed at the time these guidelines were written: Result Type Result Block

Show Result

Artist Result

Proprietary and Confidential - 2013

5.

Video Results

Video results are the most common type of YouTube search result. Here is a link to the video that is displayed when the user clicks on the title 10 Cutest Cat Moments of this video search result:

The URL of the video is http://www.youtube.com/watch?v=q1dpQKntj_w. Here is a screenshot of the landing page for this video with result components labeled:

Proprietary and Confidential - 2013

6.

Channel Results

A channel is a collection of YouTube webpages. Here are some signs that usually indicate that a particular YouTube webpage is part of a channel: A prominent black bar near the top of the page containing some or all of the following: a Subscribe button, the number of subscribers, and the number of video views Navigational tabs, such as Featured, Feed, Videos, Popular Videos, Uploaded Videos, Recent Videos, etc. Prominent display of the word channel on the page A search box titled Search Channel

All channels have at least one of these signs. With a little experience you will soon be able to recognize channels at a glance. Here are some examples. Please visit these channel pages by clicking the screenshots or the titles above the screenshots.

Channel Example 1: Lady Gaga Channel

Channel Example 2: Kitesurf Channel

Proprietary and Confidential - 2013

Channel Example 3: Pencil Drawings Channel

Channel Example 4: Puehse Tennis Channel

Examples 5 and 6 below are known as Topic channels. Topic channels are created automatically by YouTube.
Example 5: Lady Gaga Topic Channel Example 6: Dennis Quaid Topic Channel

Here is a closer look at the header of the Lady Gaga topic channel shown in Example 5:

Proprietary and Confidential - 2013

The message compiled by YouTube in this header indicates that the contents of the channel were compiled by YouTube and that the channel is therefore a YouTube channel. To see this particular channel, visit http://www.youtube.com/topic/QrFlYXe6zII/lady-gaga, and to see many more examples of topic channels, visit http://www.youtube.com/topics (YouTube Topic Channels).

For comparison, here is the header of the Lada Gaga channel shown in Example 1:

There is no message in this header indicating that the channel was compiled automatically, so it follows that the channel is manually curated, i.e., that a particular individual or organization explicitly controls what users will see on the channel. In fact, if you visit the channel at http://www.youtube.com/user/ladygagavevo?, you will see the following information in the right-hand sidebar:

This information indicates that the channel is managed by the registered YouTube user whose username is LadyGagaVEVO.

Proprietary and Confidential - 2013

7.

Playlist Results

A playlist is a list of videos arranged in a linear order. The arrangement of the videos is controlled by the registered YouTube user who owns and creates the playlist. Under the default playlist settings, videos in the playlist will play sequentially with no further action from users. Users may click videos in the playlist if they wish to watch videos in a different order. Playlist results were displayed like this at the time these guidelines were written. This is how YouTube Playlist result blocks were displayed at the time these guidelines were written:

The lower-rightmost link on the playlist block shows the approximate number of videos currently in the playlist. In the above result block, this link is titled 40 videos. Clicking on this link sends users to the List View of the playlist. It is not necessary to visit the list view to evaluate a playlist, but the screenshot below displays a portion of the List View for the playlist above.

Proprietary and Confidential - 2013

10

Most hyperlinks in a playlist result block send users to videos in the playlist. These videos are annotated with a Playlist Bottomfeeder, which is a linear display of the videos in the playlist. This is what a Playlist Bottomfeeder looked like at the time of this writing.

Please visit the playlist result above by clicking the screenshot.

Proprietary and Confidential - 2013

11

8.

Movie Results

Some movies on YouTube are free for users. Others require payment and users would not be able to view them without paying. When assigning a utility rating to such movies, please assume that the movie plays as advertised. If the title of the movie is a good fit for the query, it should get a good utility rating. You will learn more about the YouTube video rating scale in Section 10. Here are examples of free movies and movies that require payment. Movie Result Type Result Block

Free Movie Result

Notice the MOVIE label

Paid Movie Result

Notice the CHANNEL label and the 2.99 fee

Proprietary and Confidential - 2013

12

9.

Pay-Restricted Results Section added 1/10/13

Pay-restricted results are YouTube search results that cannot be viewed without first making a payment. These results have a price icon somewhere in the result block which looks like this:

Here is an example of a pay-restricted movie result:

Pay-Restricted Result Type Result Block

Pay-Restricted Movie Result

On click through, the movie result directs to an interstitial landing page that provides information about the movie (e.g. cast, reviews, etc.), and offers the option to watch a preview trailer of the movie:

Please do not confuse such preview trailers with actual movies!

Proprietary and Confidential - 2013

13

Here is an example of a pay-restricted video result:

Pay-Restricted Result Type Result Block

Pay-Restricted Video Result

On click-through, the video result directs to a YouTube page with a video player than informs users that payment is required before the video can be viewed.

Proprietary and Confidential - 2013

14

10. Rating Overview

Understanding the query and user intent As with other types of rating, you should familiarize yourself with the possible interpretations of the query and develop an understanding of the users intent. If the meaning of the query is not perfectly clear, do some research: look at the YouTube search results in the task and/or try issuing the query on google.com or wikipedia.com. You may also be able to find definitions of unfamiliar terms at urbandictionary.com. Try to imagine the various reasons a user might have issued the query on YouTube and the kind of videos that the user expects to see.

Evaluating YouTube Video Result Blocks The same rating scale will be used when evaluating all types of YouTube results; however, you will be given individual instructions for rating these four types: videos, channels, playlists, and movies.

Proprietary and Confidential - 2013

15

11. Evaluating Video Result Blocks

The usefulness of a YouTube video result is determined by the video result (the main video on the landing page) and by the users intent. If the user has a particular result (video) in mind, then that result is very useful. Results that have nothing to do with the users intent are not relevant and therefore not useful. You will assign a utility rating to YouTube video results according to the following scale:
Rating Very Useful Useful Description Use this rating when the video is not only clearly relevant to the query, but is a video that a user typing the query would expect to see and be pleased to see. Use this rating when the video has a clear connection to the query, and is a result that some users would be pleased to see but would not appeal to as wide an audience as a Very Useful video. Use this rating when the video has a connection to the query, but it would not appeal to the vast majority of users who issue the query. This rating is also appropriate for video results that serve a minor interpretation of the query. Use this rating when the video has no clear connection to the query and would evoke the response I have no idea why they are showing me this! in the vast majority of users. This rating also applies when you cannot determine the utility of the video within the allotted time. Use this rating when: N/A The video does not play. You will also check the Doesnt play (D) flag. The video is unratable because it is in a foreign language and you are unable to determine utility. You will also check the Foreign Language (F) flag.

Marginally Useful

Useless/ Off-Topic

IMPORTANT: Do NOT use this rating when the video requires payment to play. Some movie results require payment to play. For movies that do not play without payment, please assume that the movie is what it claims to be.

Important: You are not required to assign the N/A rating to foreign language videos. If you believe that a foreign language video may have some utility, you may select another rating. Here are two possible scenarios: The video doesnt need language to be appreciated. For the query [skateboarding], English (US), this
Chinese video may have some utility for English users.

The user is looking for a foreign language video. For the query [Como la Flor by Selena], English (US) this Spanish video is exactly what the user is looking for. In fact, this video is Very Useful even though it is in a foreign language.

Video Metadata and Annotations In addition to the main video, please take the following aspects of the YouTube video page into account when assigning a utility rating: Video title: Does the title accurately describe the video? Description: Does the description contain useful information or links? Annotations: Has the uploader added useful/distracting annotations?

Important: Please do not take official YouTube ads into account when assigning ratings. 16

Proprietary and Confidential - 2013

For example, if a video has lyrics in the description (metadata) like that users might find helpful, you may assign a higher rating than the video would otherwise receive. For the query [queen somebody to love], a bonus of a half-point on the utility slider is appropriate since the lyrics are for the requested song.

Proprietary and Confidential - 2013

17

On the other hand, if a video has annotations that might detract from the users experience, you may give the video a lower rating than it would otherwise receive. Without the annotations, this video would be Very Useful for the query [halloween punch prank]; however, because the uploader added so many annotations, it is only Marginally Useful. In this case, a large penalty is appropriate.

This video of the Challenger disaster has unobtrusive and informative annotations. This sort of annotation can improve the utility of a result. Use your judgment in deciding how much annotations improve a results utility. For this example, a half-point on the utility slider is an appropriate boost.

Of course, if a video is already Very Useful or Useless/Off-Topic, the rating cannot be raised or lowered. For more information about metadata, please refer to the Video Result Rating Instructions https://www.google.com/evaluation/portal/portal_files/VideoResultRating.pdf

Proprietary and Confidential - 2013

18

12. Video Result Block Rating Examples

Query [baby panda sneeze] URL
http://www.youtube.com/watch? v=FzRH3iTQPrk

Rating Very Useful

Description Video page with a video of a baby panda sneezing Video page with a video of Zinedine Zidane giving a headbutt to an Italian soccer player during the 2006 FIFA World Cup match. Video page with a video of a talking pug on the Montel Williams TV Show. Video page with a video clip of the movie The Elf. Video page with a video of the University of Southern California Marching Band playing a song from the movie Snakes on a Plane.

[zidane headbutt]

http://www.youtube.com/watch? v=zAjWi663kXc

Very Useful

[pug]

http://www.youtube.com/watch? v=Njo28CN5XuY http://www.youtube.com/watch? v=9jyCfRHumHU

Useful

[elf]

Useful

[snakes on a plane]

http://www.youtube.com/watch? v=J2IZkwgXl8M

Marginally Useful

[Star Wars]

http://www.youtube.com/watch? v=hVrIyEu6h_E

Marginally Useful

Video page with a video of Grocery Store Wars with animated fruits and vegetables playing characters from the movie, Star Wars

[hot dog]

http://www.youtube.com/watch? v=FJsQS b9RFo0

Useless/Off-Topic

Video page with a video with a recording of Elvis Presley singing Hound Dog.

[oscar ceremony]

http://www.youtube.com/watch? v=OU6EHnDAAvQ

Useless/Off-Topic

Video page with a video of a Japanese animation with a main character named Oscar.

[modern family]

http://www.youtube.com/watch? v=59twO1fJwtQ

Useless/Off-Topic

Video page with video from the TV show The View, not Modern Family

Proprietary and Confidential - 2013

19

13. Evaluating Channel Result Blocks

The first step in evaluating a channel result is to determine whether the query is channel-seeking (navigational) or non-channel-seeking (non-navigational). In other words, you should try to determine whether the query is looking for a particular channel. Note: the term navigational is used here in the context of YouTube channel results, not Web results. Channel-seeking queries. We assume that the user is looking for a YouTube channel when: a. The word channel or official is contained in the query. Here are some examples: [madonna official], [taylor swift channel], [sesame street official channel], [elton john vevo]. Note: Vevo is a music website, and channels of Vevo-affiliated artists are displayed on YouTube. b. The query is an unusual word or name which is only associated with a registered YouTube channel user. Here are some examples: [CJTClol], [kjgophish], [jimmykimmellive]. Non-channel-seeking queries. We will assume that the user is not looking for a particular YouTube channel result for all other queries.

Users who visit channel pages are typically interested in either going to a particular channel or finding videos on a particular topic. A particular channel. A user may visit Lady Gagas channel to see if Lady Gaga has posted any recent videos or to see if Lady Gaga or her fans have posted new messages. Similar content may be available on other channels, but the user is interested in this particular channel. Videos on a particular topic. A user may visit a particular channel to find a good collection of videos on a particular topic. The channel owner and the specific channel may not be important to the user it is the topic that is of interest. For example, a user interested in snowboarding may visit a channel devoted to snowboarding to peruse the collection of videos on that channel.

It is important to understand these two cases because the rating you assign to channel results should depend on the user intent.

You will assign a utility rating to channel results for channel-seeking queries according to the following scale:
Rating Very Useful Description Use this rating when you are certain that the user is looking for the particular channel. For example: [madonna channel] Use this rating when you believe it is likely or possible but not certain that the user is looking for the particular channel. For example, the query has multiple interpretations and this interpretation is likely or possible, but not dominant. Please do not use this rating when the query is channel-seeking. The channel will either be the target of the query (Very Useful), one of several likely targets of the query (Useful), or the wrong channel (Useless/Off-Topic). Use this rating when you believe the user is looking for a different channel.

Useful

Marginally Useful

Useless/Off-Topic

Proprietary and Confidential - 2013

20

For non-channel-seeking queries, please watch parts of the three videos on the channel that seem to be most relevant to the query. Based on your impression of these three videos, please assign a rating according to the following scale:
Rating Very Useful Description Please do not use this rating when the query is non-channel-seeking.

Use this rating when the following conditions are true: Useful The ratio of good-to-bad videos is very high. Very Useful and Useful videos are considered good videos. Marginally Useful and Useless/Off-Topic videos are considered bad videos. Most of the best videos on the channel are easy to find on the channel landing page; that is, the page that loads when you click the title of the channel result.

Marginally Useful

Use this rating if one or more of the above conditions is somewhat lacking. For example, the ratio of good to bad videos is even lower and/or many of the best videos are hard to find.

Useless/ Off-Topic

Use this rating when most of the videos are bad and/or the good videos are difficult to find, i.e., the good videos are not found on the landing page. For example, if you have to go to another page or if you have to use the search box on the channel page to find good videos, the good videos are difficult to find.

14. Channel Result Block Rating Examples

Query [rihanna channel] [lada gaga vevo] ChannelSeeking? Yes Yes Channel Result Rihannas Channel Lady Gagas Vevo Channel Target of the Query? Yes Yes Not a channelseeking query No Not a channelseeking query Not a channelseeking query Rating Very Useful Very Useful Explanation This channel is the target of the query, so it is Very Useful. The channel is the target of the query, so it is Useful. This channel is Useful because it is related to the artist, the ratio of good-to-bad videos is high, and the good videos are easy to find. Even though this is a high quality channel, it is not the channel the user is seeking. The user seeks a specific video which is easily found (at this writing) on the channel landing page. Other videos on the channel are Useful to the query. Even though the channel has a high ratio of good-to-bad videos, Bad Romance (the video specified in the query) is not on the landing page making it hard to find. This channel is therefore Marginally Useful.

[lady gaga]

No

Lady Gagas Vevo Channel

Useful

[chris brown official]

Yes

Madonnas Channel

Useless/ Off-Topic

[born this way]

No

Lady Gagas Vevo Channel

Useful

[bad romance]

No

Lady Gagas Vevo Channel

Marginally Useful

Proprietary and Confidential - 2013

21

Query

ChannelSeeking?

Channel Result

Target of the Query?

Rating

Explanation The Puehse twins are a pair of particularly well-known skateboarding twins. Even without their name in the query, research shows that they are the target of the query. This is their official skateboarding channel.

[skateboarding twins official]

Yes

The Puehse Twins' Skateboarding Channel

Yes

Very Useful

[puehse twins channel]

Yes

The Puehse Twins' Skateboarding Channel

Yes

Very Useful

The Puehse twins are well known for their skateboarding. This channel is the target of the query.

[puehse twins channel]

Yes

The Puehse Twins Tennis Channel

Likely or possible, but not certain

Useful

The Puehse twins have taken up tennis. This is their tennis channel. This is not the dominant interpretation of the query, but it is somewhat likely. Even though the query doesnt contain channel or official, it is navigational. There are no reasonable interpretations for the query other than this YouTube channel.

[CJTClol]

Yes

CJTClols Channel

Yes

Very Useful

[st marys college official]

St. Marys College of California Channel Yes St Marys College of Maryland Channel

Likely or possible, but not certain Likely or possible, but not certain

Useful

Useful

The query is navigational, but we cant tell which St. Marys channel the user is looking for. Channels for both colleges are Useful.

[Yosemite]

No

Yosemite National Park Topic Channel

Not a channelseeking query

Useful

This topic channel is Useful because it is related to the query, the ratio of good-tobad videos is high, and the good videos are easy to find.

[dogs]

No

Gray Wolf Topic Channel

Not a channelseeking query

Marginally Useful

Even though the videos are high quality, they have just a marginal connection to the query. The vast majority of users who issue the query are looking for videos of household dogs, not gray wolves that live in the wild.

Please note: Even though topic channels are created differently, they are rated in the same way as other channel results.

Proprietary and Confidential - 2013

22

15. Evaluating Playlist Result Blocks

When evaluating playlist results, you are not required to watch all of the videos in the playlist. You will form an opinion of the videos in the playlist by watching parts of the first three videos that play when you click on the URL in the rating task. Please base your rating on these three videos: The video that plays when you click the URL in the rating task, and The following two videos that appear in the playlist bottomfeeder.

Please ignore the remaining videos in the playlist. The following diagram shows an example of a video result block and indicates on the playlist bottomfeeder the videos you should watch before assigning your rating.

It may occasionally happen that you are sent to a video that does not have a playlist bottomfeeder when you click the title of the playlist result block. If this happens, please do both of the following: Flag the playlist as Doesnt play to indicate that there is a problem with the playlist bottomfeeder Watch the three videos linked from the result block and assign a utility rating according to the guidelines below.

It may also happen that one or more of the three videos you should watch doesnt load or fails to play. If this happens, please do the following: Flag the playlist as Doesnt play to indicate there is a problem with at least one of the videos. Watch the other videos and assign a rating according to the guidelines below.

Important: Please do not watch any extra videos to compensate for videos that do not load.

Proprietary and Confidential - 2013

23

Evaluating Playlist Results The first step in evaluating a playlist result is to determine whether the query is playlist-seeking or non-playlistseeking. Playlist-seeking queries. When the query includes words like playlist, songs, top hits, or videos, we will assume the user is looking for a playlist result. Notice that songs, top hits, and videos are pluralized, which indicates the user is looking for more than one song or video. Non-playlist-seeking queries. We will assume that the user is not looking for a playlist result when the above words (or similar types of words) are not included in the query.

You will assign a utility rating to playlist results for playlist-seeking queries according to the following scale:
Rating Very Useful Useful Marginally Useful Useless/Off-Topic Description Use this rating when all or most of the three videos you watch are Very Useful for the query. Use this rating when all or most of the three videos you watch are Useful for the query. Use this rating when all or most of the three videos you watch are only Marginally Useful for the query. Use this rating when all or most of the three videos you watch are Useless/Off-Topic to the query and most users would wonder why they are being shown the playlist result.

You will assign a utility rating to playlist results for non-playlist-seeking queries according to the following scale:
Rating Very Useful Useful Marginally Useful Description Please do not use this rating when the query is non-playlist-seeking. Use this rating when all or most of the three videos you watch are Very Useful for the query. Use this rating when all or most of the three videos you watch are Useful for the query. Use this rating when all or most of the three videos you watch are Marginally Useful or Useless/OffTopic for the query.

Useless/Off-Topic

Please note that the rating standards are high when the query is non-playlist-seeking. To be rated Useful, all or most of the three individual videos you watch must be Very Useful. To be Marginally Useful, all or most of the three videos you watch must be Useful.

Proprietary and Confidential - 2013

24

16. Playlist Result Block Rating Examples

Query Playlist Seeking? Playlist Result Madonna YouTube Playlist Result 1 Description The main video is Madonnas Like a Virgin video. The next two videos are her Express Yourself and Erotica videos. Rating Explanation Since the query has playlist in it, we know it is playlist-seeking. The main video and the next two videos are Madonna songs. Most users would consider these videos Very Useful or Useful. Since the query has songs in it, we know it is playlist-seeking. The main video and the next two videos are Madonna songs. Most users would consider these videos Very Useful or Useful. For non-playlist-seeking queries, Useful is the highest rating that can be assigned.

[Madonna playlist]

Yes

Very Useful

[Madonna songs]

Yes

Madonna YouTube Playlist Result 1

The main video is Madonnas Like a Virgin video. The next two videos are her Express Yourself and Erotica videos. The main video is Madonnas Like a Virgin video. The next two videos are her Express Yourself and Erotica videos. The main video is a video of an interview Madonna did on a TV show. The second video is a promotional documentary of W.E., a movie directed by Madonna. The third video is a trailer for the W.E. movie. The main video is a video of an interview Madonna did on a TV show. The second video is a promotional documentary of W.E., a movie directed by Madonna. The third video is a trailer for the W.E. movie.

Very Useful

[Madonna]

No

Madonna YouTube Playlist Result 1

Useful

[Madonna songs]

Yes

Madonna YouTube Playlist Result 2

Useless/ Off-Topic

Even though the query is playlistseeking and the result is a playlist, this particular playlist result is Useless/Off-Topic for the query. User intent is to watch videos of Madonnas songs, not watch interviews, documentaries, etc. Most users would be looking for music videos and not interviews, documentaries, etc. All of these videos are Marginally Useful for the user intent, and because the query is not playlist-seeking, the rating is demoted one level. Since the query has songs in it, we know it is playlist-seeking. The main video and the next two videos are songs from the childrens show Sesame Street. Most users would consider these videos Very Useful or Useful.

[Madonna]

No

Madonna YouTube Playlist Result 2

Marginally Useful

[sesame street songs]

Yes

Sesame Street Songs Playlist Result

The main video and the next two videos are music videos from the childrens show Sesame Street.

Very Useful

[sesame street]

No

Sesame Street Songs Playlist Result

The main video and the next two videos are music videos from the childrens show Sesame Street.

Useful

For non-playlist-seeking queries, Useful is the highest rating that can be assigned.

Proprietary and Confidential - 2013

25

17. Evaluating Movie Result Blocks

The first step in evaluating a movie result is to determine what the query is looking for. In some cases, the query is looking for a particular movie. In other cases, the query is looking for a celebrity or a subject. You must try to imagine user intent when assigning a rating to movie results. You will assign a utility rating to movie results according to the following scale:
Rating Very Useful Useful Description Use this rating when you are certain that the user is looking for the movie. Use this rating when the movie has a clear connection to the query, and is a result that some users would be pleased to see but would not appeal to as wide an audience as a Very Useful movie. Use this rating when the movie has a connection to the query, but it would not appeal to the vast majority of users who issue the query. This rating is also appropriate for movie results that serve a minor interpretation of the query. Use this rating when the movie has no clear connection to the query and would evoke the response I have no idea why they are showing me this! in the vast majority of users. This rating also applies when you cannot determine the utility of the movie within the allotted time. Use this rating when: The movie does not play. You will also check the Doesnt play (D) flag. The video is unratable because it is in a foreign language and you are unable to determine utility. You will also check the Foreign Language (F) flag.

Marginally Useful

Useless/Off-Topic

N/A

IMPORTANT: Do NOT use this rating when the movie requires payment to play. For movies that do not play without payment, please assume that the movie is what it claims to be.

18. Movie Result Block Rating Examples

Query URL Rating Description This movie matches the query. It is what the user is looking for. [fletch] http://www.youtube.com/ movie?v=-ssId2r7F1I Very Useful This movie requires payment. Although we cant watch the movie without paying, we will assume that the movie is what it claims to be. This movie matches the query. It is what the user is looking for. Very Useful This movie requires payment. Although we cant watch the movie without paying, we will assume that the movie is what it claims to be. This movie has a connection to the query, but it would not appeal to the vast majority of users who issue the query. Most users who issue this query are not looking for a movie; they want to watch videos of the band The Police.

[gone in 60 seconds]

http://www.youtube.com/ movie?v=vt1AFkNitcU&o b=av1n&feature=mv_sr

[police]

http://www.youtube.com/ movie?v=QuHMBFTBD5 s

Marginally Useful

Proprietary and Confidential - 2013

26

19. Evaluating Pay-Restricted Results Section added 1/10/13

If the result requires payment, you are not required to pay and are not required to watch. We still ask that you rate the block, even without the ability to view the video. Use the title, snippet, and thumbnail to give your best guess. Do not assign the "Doesn't play" flag to pay-restricted results. (The Doesnt play flag signals to us that there may be a problem with the result, but pay-restricted results typically function as we intend them to function.)

20. Pay-Restricted Result Block Rating Examples Section added 1/10/13

Query [adventure time]

Pay-Restricted Result Block

Rating Description

Useful The user seeks episodes or clips from the Adventure Time TV show. The video uploader (CartoonNetworkEps) seems like the official owner of this TV show and this seems like an episode from the show. Many users would find this result appealing if it were free, so it should be rated Useful.

Query

[adventure time jiggler]

Pay-Restricted Result Block

Rating Description

Very Useful The user seeks a particular episode of this TV show. This is exactly what the user is looking for.

Query

[adventure time jiggler]

Pay-Restricted Result Block

Rating Description

Marginally Useful The user is looking for a particular episode of this TV show. This is the wrong episode, so it will have limited appeal.

Proprietary and Confidential - 2013

27

Query

[adventure time jiggler free]

Pay-Restricted Movie Result Block

Rating Description

Useless to Marginally Useful

The user seeks a particular episode of this TV show that can be viewed without charge. This payrestricted result has limited appeal.

21. Assigning Flags to YouTube Result Blocks

You will assign the following flags as needed: D = Doesn't play, P = Porn/offensive, S = Spam, F = Foreign Language, O = Old/Stale, A = Authoritative/Official

Assigning flags to video results

Doesnt play: Use this flag when the video does not play for any reason, e.g. taken down, marked as private, prohibited for the locale, etc. Do NOT use this flag for paid results, which typically function as intended after payment.

Porn/offensive: Use this flag when the content of the video result is offensive or shocking because of sexrelated content, violence, bad language, etc.

Spam: Use this flag when the content of the video result is misleading and/or commercial in nature. You may also apply this flag if text, links, or other forms of advertising are superimposed over a video, and their presence negatively affects user experience. This will be a judgment call on your part but, in general, a single link back to the source of the video is not considered to be Spam.

Foreign Language: Use this flag for videos that would be regarded as foreign by most users in your locale. In general, the N/A rating is appropriate for foreign videos unless you think the video has some utility despite being foreign. In that case, please assign a rating from the rating scale.

Old/Stale: Use this flag for videos that are stale for this query. A video is stale if all of the following are true: o There are at least two interpretations of the query o You believe that the user intends the more recent interpretation o The result pertains to an older interpretation

Authoritative/Official: A YouTube video is Official or Authoritative if you believe that the registered YouTube user who posted the video is an official source for the video. Here are some examples:

Proprietary and Confidential - 2013

28

Spam Example 1 Query: [house of anubis s02 episode 1]

Explanation

A result like this should be flagged as Spam. Someone uploaded this video with the sole intent of getting users to click the link highlighted in red. Even if the link provides the user what the spammer promises, this result is spam. If you see spammy links in video metadata, please flag the result. Even if the spam link works and provides the user with the promised episode, the result is still Useless/Off-Topic because the static image video has no utility.

Spam Example 1 Query: [best fails]

Explanation

Even without the spam link in the video description, this video deserves a Spam flag because of the obtrusive commercial annotations (big red blocks of text in the video). This result also deserves a utility demotion because of these distracting annotations. In general, you should use your judgment when deciding how severely spam and distracting annotations affect result utility. A one-half to one full rating demotion would be appropriate in this case. The result would be considered Marginally Useful for the query [best fails].

Proprietary and Confidential - 2013

29

Old/Stale Example Query: [tsunami]

Explanation

For the query [tsunami], this video about the 2004 Indian Ocean tsunami should be marked as Old/Stale because most users who issue this query are probably interested in more recent tsunamis.

Old/Stale Example Query: [world cup final]

Explanation

For the query [world cup final], this video about the 2002 World Cup Final should be marked as Old/Stale because most users who issue this query are probably interested in the most recent World Cup final game.

Proprietary and Confidential - 2013

30

Authoritative/Official Example 1 Query: [Robyn]

Explanation

RobynVEVO is the official source for music videos featuring the artist Robyn.

Authoritative/Official Example 2 Query: [barack obama]

Explanation

whitehouse is the official source for videos featuring the US President in his official capacity.

Proprietary and Confidential - 2013

31

Assigning flags to channel and playlist results Authoritative/Official: Assign this flag if the channel is official or if the playlist is created by the owner of a channel that is official for the query. Do not assign this flag to topic channels. Flags other than the Authoritative/Official flag: channel or playlist deserve the flag. Assign the appropriate flag if ANY of the results you see on the

22. Reporting Results with Duplicate Landing Pages

In addition to assigning utility ratings to each display block result, we would like you to tell us whether two results are duplicates (dupes).

Pre-Identified Duplicates
Some duplicate results are automatically detected and will be pre-identified for you. They will be highlighted by a grey or green background, and annotated by either "Same as..." text right below the flags and above the result block. This is what a pre-identified Same as result block look like: Same as pre-identified duplicate result

Please note: Results annotated by Same as may not be given different ratings/flags/comments. Their utility ratings, page quality ratings, flags, and comments will be automatically transferred to each other. You cannot un-check dupes that have been automatically detected and pre-identified.

Rater-Identified Duplicates
We would like your help identifying duplicate results that have not been automatically detected. Encountering a list of identical or very similar results on the YouTube search results page is generally a bad experience for users. Use your judgment when applying this definition: two video results are duplicates if they have similar or identical content and the user who issued the query would not find the second video at all engaging after having enjoyed the first to its fullest extent. A video result should NOT be marked as a duplicate of a playlist or channel result.

Proprietary and Confidential - 2013

32

Reporting Duplicate Results

When you notice that the results in two or more display blocks are duplicates, please click on the Report Dupe button of one of the results. The current result (the result you clicked on) will be highlighted by a thick solid red border and the button's name will change to Select Dupes. You can then check results that are duplicates of the current result, and the checked results will be highlighted by a red dotted border. The duplicate results that you checked will be annotated by "Dupe of..." text right below the flags and above the display block. After selecting all dupes, please click the red Select Dupes button to return to the normal rating mode. The button's name will change back to Report Dupe, and you will be able to report other sets of dupes (if there are any). If you change your mind, you can always un-check a result. Please note that rater-identified duplicate results annotated by "Dupe of..." may be given different utility ratings/flags/comments. Please identify dupes both within the same side and across sides. Even for cross-side results, you should still ask yourself the question "Would you want to see both results if they were returned by the same search engine?" Here are some examples: Dupes QUERY: [Princess Mononoke Trailer] URL 1: http://www.youtube.com/watch?v=oZ4pxURy5-I URL 2: http://www.youtube.com/watch?v=pkWWWKKA8jY Reason: These two video results are duplicates because they are movie trailers that differ only in aspect ratio. Even though they were uploaded by different YouTube users and their run times are slightly different, they are duplicates. Users would not want to see both results returned by YouTube. Not Dupes QUERY: [Por una cabeza] URL 1: http://www.youtube.com/watch?v=7ydRwCf9Qyw URL 2: http://www.youtube.com/watch?v=dBHhSVJ_S6A&feature=related Reason: These videos show actors dancing the tango to Por una Cabeza from two different movies (True Lies and Scent of a Woman). Even though the song is the same, the videos are very different. Users could benefit by YouTube returning both videos for the query.

23. Assigning an Overall Side by Side Score (if requested)

When you are done assigning ratings and flags and flagging pages with duplicate landing pages, please decide which side is better according to the SxS Guidelines. Reminder: You will only be asked to assign an Overall Side by Side score in some Block Utility tasks.

Proprietary and Confidential - 2013

33