You are on page 1of 221

YQL Guide

Abstract This guide describes the Yahoo! Query Language (YQL) Web Service, which enables you to access Internet data with SQL-like commands. This guide is for software developers who are famliar with Web applications that call Web services to retrieve data in XML or JSON format. Experience with SQL, MySQL, or Yahoo! Pipes is helpful, but not required. For an overview of the features of the latest YQL release, refer to the YQL Release Notes.1 We welcome your feedback. Have a comment or question about this document? Let us know in the YDN Forum for Y!OS Documentation2.

1 2

http://developer.yahoo.com/yql/releasenotes/ http://developer.yahoo.net/forum/index.php?showforum=64

Table of Contents
Introducing YQL ................................................................................................................. 1 1. Overview ................................................................................................................ 1 What is YQL? ..................................................................................................... 1 Why Use YQL? ................................................................................................... 1 Usage Information and Limits ................................................................................ 2 Limiting Access to Content Provider Data ................................................................ 2 Blocking HTML Data Scraping from YQL ....................................................... 2 Blocking Non-HTML from YQL .................................................................... 3 Rate Limiting by IP Address .......................................................................... 3 Internationalization Support ................................................................................... 4 Character Encoding ...................................................................................... 4 2. YQL Tutorials and Code Examples .............................................................................. 5 Introduction ........................................................................................................ 5 Prerequisites ....................................................................................................... 5 YQL Tutorials ..................................................................................................... 5 First YQL Application ................................................................................. 6 Creating YQL Open Data Tables ..................................................................... 9 Executable JavaScript in Open Data Tables ..................................................... 12 YQL Code Examples .......................................................................................... 20 Prerequisites ............................................................................................. 20 Making YQL Queries with JavaScript ............................................................ 20 Making YQL Queries with PHP .................................................................... 22 YQL Social Application .............................................................................. 24 YQL INSERT: WordPress Open Application ................................................... 28 Getting Updates with YQL ........................................................................... 31 Data Scraping with YQL ............................................................................. 33 Sample Open Data Tables .................................................................................... 35 YQL Screencasts ............................................................................................... 35 YQL Slideshows ................................................................................................ 36 Additional Tutorials and Code Examples ................................................................ 36 Additional References and Resources .................................................................... 37 Using YQL and Open Data Tables ........................................................................................... 1 1. Overview for YQL Users ........................................................................................... 1 How to Run YQL Statements ................................................................................. 1 The Two-Minute Tutorial ...................................................................................... 1 YQL Web Service URLs ....................................................................................... 2 YQL Query Parameters ................................................................................. 3 YQL Query Aliases ...................................................................................... 3 Summary of YQL Statements ................................................................................. 5 Authorization ...................................................................................................... 5 2. SELECT Statement .................................................................................................. 6 Introduction to SELECT ....................................................................................... 6 Syntax of SELECT .............................................................................................. 6 Specifying the Elements Returned (Projection) .......................................................... 7 Filtering Query Results (WHERE) .......................................................................... 7 Remote Filters ............................................................................................ 7 Local Filters ............................................................................................... 8 Combining Filter Expressions (AND, OR) ........................................................ 9 Joining Tables With Sub-selects ............................................................................ 10 Paging and Table Limits ...................................................................................... 10 Remote Limits ........................................................................................... 11

Yahoo! Developer Network

September 8, 2011

YQL Guide

Local Limits ............................................................................................. 11 Sort and Other Functions ..................................................................................... 12 Remote and Local Processing ............................................................................... 12 Example With Remote and Local Steps .......................................................... 13 Summary of Remote and Local Controls ........................................................ 14 GUIDs, Social, and Me ....................................................................................... 14 Variable Substitution in the GET Query String ......................................................... 14 Extracting HTML Content With XPath ................................................................... 15 3. Using the Query Builder .......................................................................................... 17 YQL Query Builder Overview .............................................................................. 17 Query Builder Usage .......................................................................................... 18 Loading a Query ........................................................................................ 18 Entering Values ......................................................................................... 19 4. INSERT, UPDATE, and DELETE (I/U/D) Statements ................................................... 20 Introduction to INSERT, UPDATE, and DELETE (I/U/D) Statements .......................... 20 Bindings Required for I/U/D ................................................................................ 20 JavaScript Methods Available to I/U/D ................................................................... 22 Syntax of I/U/D ................................................................................................. 22 Syntax of INSERT ..................................................................................... 22 Syntax of UPDATE .................................................................................... 22 Syntax of DELETE .................................................................................... 23 Limitations of I/U/D ........................................................................................... 23 Open Data Table Examples of I/U/D ...................................................................... 24 Bit.ly Shorten URL (INSERT) ...................................................................... 24 WordPress Post (INSERT) ........................................................................... 25 5. Response Data ....................................................................................................... 34 Supported Response Formats ............................................................................... 34 JSONP-X: JSON envelope with XML content ......................................................... 34 Structure of Response ......................................................................................... 36 Information about the YQL Call ............................................................................ 36 XML-to-JSON Transformation ............................................................................. 37 JSON-to-JSON Transformation ............................................................................ 38 Errors and HTTP Response Codes ......................................................................... 39 6. Using YQL Open Data Tables ................................................................................... 41 Overview of Open Data Tables ............................................................................. 41 Invoking an Open Data Table Definition within YQL ................................................ 41 Invoking a Single Open Data Table ................................................................ 41 Invoking Multiple Open Data Tables .............................................................. 42 Invoking Multiple Open Data Tables as an Environment .................................... 42 Working with Nested Environment Files ......................................................... 43 Setting Key Values for Open Data Tables ................................................................ 44 Using SET to Hide Key Values or Data .......................................................... 45 Creating YQL Open Data Tables ............................................................................................. 1 1. Creating YQL Open Data Tables ................................................................................. 1 Before You Begin ................................................................................................ 1 Open Data Tables Reference .................................................................................. 1 table element .............................................................................................. 2 meta element .............................................................................................. 2 select / insert / update / delete elements ............................................................ 3 function Element ....................................................................................... 37 url element ................................................................................................. 5 execute element ........................................................................................... 5 key / value / map elements ............................................................................. 6 paging element ............................................................................................ 9

Yahoo! Developer Network

September 8, 2011

YQL Guide

YQL Streaming ................................................................................................. 11 Configuring Open Data Tables to Use Streaming .............................................. 11 Using Open Data Tables Configured for Streaming ........................................... 12 Debugging Open Data Tables and YQL Network Calls .............................................. 12 Enabling Logging ...................................................................................... 13 Viewing Logs ............................................................................................ 13 Open Data Table Examples .................................................................................. 14 Flickr Photo Search .................................................................................... 14 Digg Events via Gnip .................................................................................. 15 Open Data Tables Security and Access Control ........................................................ 16 Batching Multiple Calls into a Single Request ......................................................... 17 Troubleshooting ................................................................................................. 17 2. Executing JavaScript in Open Data Tables ................................................................... 19 Introduction ...................................................................................................... 19 Features and Benefits .................................................................................. 19 Ensuring the Security of Private Information ........................................................... 20 JavaScript Objects, Methods, and Variables Reference ............................................... 20 y Global Object ......................................................................................... 21 request Global Object ................................................................................. 30 rest Object ................................................................................................ 30 response Global Object ............................................................................... 37 result Object ............................................................................................. 37 internal Global Object ................................................................................. 38 Global Variables ........................................................................................ 42 JavaScript and E4X Best Practices for YQL ............................................................ 43 Paging Results ........................................................................................... 43 Including Useful JavaScript Libraries ............................................................. 43 Using E4X within YQL ............................................................................... 44 JavaScript Logging and Debugging ............................................................... 46 Executing JavaScript Globally ...................................................................... 46 Making Asynchronous Calls with JavaScript Execute ........................................ 47 Preventing Lossy JSON Results .................................................................... 50 Examples of Open Data Tables with JavaScript ........................................................ 50 Hello World Table ...................................................................................... 51 Yahoo! Messenger Status ............................................................................. 51 OAuth Signed Request to Netflix .................................................................. 52 Request for a Flickr frob .............................................................................. 54 Celebrity Birthday Search using IMDB .......................................................... 55 Shared Yahoo! Applications ......................................................................... 58 CSS Selector for HTML .............................................................................. 60 Execution Rate Limits ......................................................................................... 61 3. Creating YQL Customized Functions .......................................................................... 63 Introduction ...................................................................................................... 63 Types of Functions ............................................................................................. 63 How to Use a Customized Function ....................................................................... 63 function Element ............................................................................................... 64 Stream Functions ............................................................................................... 64 Example Stream Function ............................................................................ 65 Reduce Functions ............................................................................................... 66 Reference ......................................................................................................... 68 4. Using Hosted Storage with YQL ................................................................................ 69 Introduction ...................................................................................................... 69 About YQL Hosted Storage ......................................................................... 69 Storage Limits and Requirements .................................................................. 69

Yahoo! Developer Network

September 8, 2011

YQL Guide

Storing New Records .......................................................................................... 70 Storing a New Record using Text .................................................................. 70 Storing a New Record using Data from an URL ............................................... 71 Storing a New Named Record using Data from an URL .................................... 71 Using YQL to Read, Update, and Delete Records ..................................................... 71 Accessing Records using YQL ...................................................................... 71 Deleting Records using YQL ........................................................................ 72 Updating Records using YQL ....................................................................... 72 Using Records within YQL .................................................................................. 72 Using Hosted Environment Files ................................................................... 73 Using Hosted YQL Open Data Tables ............................................................ 73 Including Hosted JavaScript ......................................................................... 73 YQL for Yahoos .................................................................................................................. 1 1. Overview ................................................................................................................ 1 Audience ........................................................................................................... 1 Why Use YQL? ................................................................................................... 1 Features of Internal Open Data Tables ............................................................. 1 YQL as Your Web Service ............................................................................. 1 YQL Request Flow .............................................................................................. 2 Authorization .............................................................................................. 3 Request Parsing ........................................................................................... 3 2. Getting Started ......................................................................................................... 6 Requirements ...................................................................................................... 6 On-Boarding ....................................................................................................... 6 Accessing Internal Data Tables ............................................................................... 7 Setup ......................................................................................................... 7 Example PHP Source Code ........................................................................... 8 Creating an External Open Data Table Containing Public Data ..................................... 9 Summary of Steps ........................................................................................ 9 Convert the Table Definition to a Core Table ................................................... 10 Creating an External Open Data Table Containing Internal Data .................................. 10 Summary of Steps ...................................................................................... 10 Get Paranoid Approval ................................................................................ 11 Define a Table With Internal Data ................................................................. 11 Internal YQL Endpoints for Development ....................................................... 12 Rate Limiting for Tables .............................................................................. 13 Authenticated Data Access Through YQL ............................................................... 16 Introduction .............................................................................................. 16 Setup ....................................................................................................... 16 Examples ................................................................................................. 16 3. Using YQL to Connect to a Database .......................................................................... 21 Requirements .................................................................................................... 21 Making YQL Queries on the MySQL Database ........................................................ 21 Example YQL Statements and Responses ............................................................... 22 4. Java Execute in Data Tables ...................................................................................... 23 Why Java Instead of JavaScript? ........................................................................... 27 Requirements .................................................................................................... 28 Creating a YQL Data Table with Java Execute ........................................................ 29 5. Accessing Cookies from JavaScript Execute ................................................................ 27 Use Cases ......................................................................................................... 27 Creating B-Cookies ............................................................................................ 27 Getting Cookies ................................................................................................. 28 Setting Cookies ................................................................................................. 28 Examples ......................................................................................................... 29

Yahoo! Developer Network

September 8, 2011

YQL Guide

6. YQL Performance ................................................................................................... 31 Latency versus Throughput .................................................................................. 31 Monitoring YQL Performance .............................................................................. 31 Caching and Storage ........................................................................................... 31 Optimizing YQL Calls ........................................................................................ 31 Make Parallel Calls .................................................................................... 31 Use Timeouts ............................................................................................ 31 Check for Cached Queries .......................................................................... 32 Use XML ................................................................................................. 32 Use YQL Caches and Storage ....................................................................... 32 7. YQL Extended Schema ............................................................................................ 33 Introduction ...................................................................................................... 33 Reference ......................................................................................................... 33 table Element ............................................................................................ 33 javaExecute Element .................................................................................. 34 connect Element ........................................................................................ 34 select | insert | update | delete Element ............................................................ 35 javaExecute Element (Within Bindings) ......................................................... 36 ratelimit Element ....................................................................................... 36 item Element ............................................................................................. 36 function Element ....................................................................................... 37 Example Table Definitions ................................................................................... 37 A. FAQ .................................................................................................................... 38

Yahoo! Developer Network

September 8, 2011

Introducing YQL

Introducing YQL
Abstract This part of the YQL Guide introduces the Yahoo! Query Language (YQL). It discusses YQL uses, benefits, rate limits, and how content providers can limit access to YQL users. For an overview of the features of the latest YQL release, refer to the YQL Release Notes.3 Looking for more docs? See the YOS Documentation4 landing page. We welcome your feedback. Have a comment or question about this document? Let us know in the YDN Forum for YOS Documentation5.

3 4

http://developer.yahoo.com/yql/releasenotes/ /yos 5 http://developer.yahoo.net/forum/index.php?showforum=64

Table of Contents
1. Overview ........................................................................................................................ 1 What is YQL? ............................................................................................................. 1 Why Use YQL? ........................................................................................................... 1 Usage Information and Limits ........................................................................................ 2 Limiting Access to Content Provider Data ........................................................................ 2 Blocking HTML Data Scraping from YQL ............................................................... 2 Blocking Non-HTML from YQL ............................................................................ 3 Rate Limiting by IP Address .................................................................................. 3 Internationalization Support ........................................................................................... 4 Character Encoding .............................................................................................. 4 2. YQL Tutorials and Code Examples ...................................................................................... 5 Introduction ................................................................................................................ 5 Prerequisites ............................................................................................................... 5 YQL Tutorials ............................................................................................................. 5 First YQL Application ......................................................................................... 6 Creating YQL Open Data Tables ............................................................................. 9 Executable JavaScript in Open Data Tables ............................................................. 12 YQL Code Examples .................................................................................................. 20 Prerequisites ..................................................................................................... 20 Making YQL Queries with JavaScript .................................................................... 20 Making YQL Queries with PHP ............................................................................ 22 YQL Social Application ...................................................................................... 24 YQL INSERT: WordPress Open Application ........................................................... 28 Getting Updates with YQL ................................................................................... 31 Data Scraping with YQL ..................................................................................... 33 Sample Open Data Tables ............................................................................................ 35 YQL Screencasts ....................................................................................................... 35 YQL Slideshows ........................................................................................................ 36 Additional Tutorials and Code Examples ........................................................................ 36 Additional References and Resources ............................................................................ 37

Yahoo! Developer Network

iii

September 8, 2011

List of Figures
2.1. YQL News Application ................................................................................................... 7 2.2. YQL Sushi Restaurant Finder Application ........................................................................... 8

Yahoo! Developer Network

iv

September 8, 2011

Chapter 1. Overview
In this Chapter: What is YQL? [1] Why Use YQL? [1] Usage Information and Limits [2] Limiting Access to Content Provider Data [2] Internationalization Support [4]

What is YQL?
The YQL Web Service enables applications to query, filter, and combine data from different sources across the Internet. YQL statements have a SQL-like syntax, familiar to any developer with database experience. The following YQL statement, for example, retrieves a list of cat photos from Flickr: SELECT * FROM flickr.photos.search WHERE text="cat" To access the YQL Web Service, a Web application can call HTTP GET, passing the YQL statement as a URL parameter, for example: http://query.yahooapis.com/v1/public/yql?q=SELECT tos.search WHERE text="Cat" * FROM flickr.pho-

When it processes a query, the YQL Web Service accesses a datasource on the Internet, transforms the data, and returns the results in either XML or JSON format. YQL can access several types of datasources, including Yahoo! Web Services, other Web services, and Web content in formats such as HTML, XML, RSS, and Atom.

Why Use YQL?


The YQL Web Service offers the following benefits: Because it resembles SQL, the syntax of YQL is already familiar to many developers. YQL hides the complexity of Web service APIs by presenting data as simple tables, rows, and columns. YQL includes pre-defined tables for popular Yahoo! Web services such as Flickr, Social, MyBlogLog, and Search. YQL can access services on the Internet that output data in the following formats: HTML, XML, JSON, RSS, Atom, and microformat. YQL is extensible, allowing you to define Open Data Tables [41] to access datasources other than Yahoo! Web Services. This feature enables you to mash up (combine) data from multiple Web services and APIs, exposing the data as a single YQL table. You can choose either XML or JSON for the format [34] of the results returned by requests to YQL.

Yahoo! Developer Network

September 8, 2011

Overview

YQL sub-selects [10] enable you to join data from disparate datasources on the Web. YQL returns the data in a structured document, with elements that resemble rows in a table. With YQL, you can filter [7] the data returned with an expression that is similar to the WHERE clause of SQL. When processing data from large tables, you can page [10] through the query results. The YQL Console1 enables you to run YQL statements interactively from your browser. The console includes runnable sample queries so that you can quickly learn YQL. For a quick introduction to the console, see the Two-Minute Tutorial [1].

Usage Information and Limits


The following information describes the use, performance, dependencies, and limits of the YQL Web service. If you have additional questions, please read the YQL Terms of Service2 or send an email to <yql-questions@yahoo-inc.com>. Usage Information: YQL can be used for commercial purposes. If we're going to shut down YQL, we will give you at least 6 months notice with an announcement on YDN and in our forum. YQL has a performance uptime target of over 99.5%. YQL relies on the correct operation of the Web services and content providers it accesses. Rate Limits: Per application limit (identified by your Access Key): 100,000 calls per day. Per IP limits: /v1/public/*: 1,000 calls per hour; /v1/yql/*: 10,000 calls per hour. All rates are subject to change. YQL rate limits are subject to the rate limits of other Yahoo! and 3rd-party Web services.

Limiting Access to Content Provider Data


Content or API providers can opt out or restrict YQL access to their data by following the instructions in the sections below.

Blocking HTML Data Scraping from YQL


YQL uses the robots.txt file on your server to determine the Web pages accessible from your site. YQL uses the user-agent "Yahoo Pipes 2.0" when accessing the robots.txt file and checks it for allows/disallows from this user agent. If the robots.txt check does prevent YQL from accessing your content, it will then fetch the target page using a different user agent:

1 2

http://developer.yahoo.com/yql/console/ http://info.yahoo.com/legal/us/yahoo/yql/yql-4307.html

Yahoo! Developer Network

September 8, 2011

Overview

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14 Therefore, to deny YQL access to your content, simply add "Yahoo Pipes 2.0" to the relevent parts of your robots.txt. For example: User-agent: Yahoo Pipes 2.0 Disallow: / Another approach is to block YQL on your Web server. For example, in Apache, add this to your virtual host block in httpd.conf: SetEnvIfNoCase User-Agent "Yahoo Pipes" noYQL <Limit GET POST> Order Allow,Deny Allow from all Deny from env=noYQL </Limit>

Blocking Non-HTML from YQL


YQL fetches content from URLs when requested by a developer. Because YQL is not a Web crawler, it does not follow the robots exclusion protocol for non-HTML data, such as XML or CSV, from a site. To stop YQL from accessing any content on your site, block the YQL user-agent (Yahoo Pipes 2.0) on your Web server. For example, on Apache servers, add this rule to your virtual host block in httpd.conf: SetEnvIfNoCase User-Agent "Yahoo Pipes" noYQL <Limit GET POST> Order Allow,Deny Allow from all Deny from env=noYQL </Limit>

Rate Limiting by IP Address


YQL allows APIs to accurately use IP-based rate limits that will track and count on the YQL developer's IP address, rather than the IP addresses of shared proxy servers that YQL uses to access content on the Web. For outgoing requests to external content and API providers, YQL determines the last valid client IP address connecting to its Web service and then ensures this is the first IP address in the X-FORWARDED-FOR HTTP header. For example, in the X-FORWARDED-FOR HTTP header below, the request arriving at YQL came from the 1.2.3.4 IP address. IP-rate limiters should use this value rather than the IP addresses of YQL proxy servers. X-FORWARDED-FOR: 1.2.3.4, 5.6.7.8, 9.10.11.12 We also set the CLIENT-IP HTTP header to this IP address. For example:

Yahoo! Developer Network

September 8, 2011

Overview

CLIENT-IP: 1.2.3.4

Note
Because these headers are "unsigned," they can be spoofed. Therefore, providers should only use these headers if the proxy setting them is trusted. The IP addresses of the proxy hosts that should be trusted can be found at http://developer.yahoo.com/yql/proxy.txt. This file will be updated as our proxy hosts change.

Internationalization Support
Character Encoding
YQL supports most of the character sets in the IANA Character Sets Registery3. YQL uses the HTTP header Content-Type in the request to determine the character encoding for the response body. If no character encoding is specified, YQL uses the default UTF-8. The YQL statement can also determine the character encoding for the body with the key charset. If the character encoding is specified in both places, the character set specified by charset has precedence. For example, to request YQL use ISO/IEC 8859-1 to encode the response body, do one of the following: In your request, set the HTTP header Content-Type as shown below: Content-Type: text/html; charset=iso-8859-1 In the YQL statement, specify the character set with the key charset as shown below: select * from html where url='http://example.com' and charset='iso8559-1'

Note
The YQL built-in function sort only correctly sorts results in English.

http://www.iana.org/assignments/character-sets

Yahoo! Developer Network

September 8, 2011

Chapter 2. YQL Tutorials and Code Examples


Introduction
This chapter provides the following: YQL Tutorials [5]: Step-by-step instructions on how to use YQL. YQL Code Examples [20]: PHP and JavaScript code with brief explanations. Sample Open Data Tables [35]: Open Data Table examples that showcase the various capabilities of YQL. YQL Screencasts [35]: Video screencasts that discuss YQL features and usage. YQL Slideshows [36]: Slideshow presentations about YQL and how it can help developers. Additional Tutorials and Code Examples [36]: More advanced tutorials and code examples showing specialized uses for YQL. Additional References and Resources [37]: Additional references and resources for YQL, such as libraries, handouts, blog posts, and demos.

Prerequisites
Two-Minute Tutorial [1] Have a Web server that can be accessed over the Internet. Be familiar with the following technologies: SQL HTML/XHTML JavaScript Web feeds

YQL Tutorials
First YQL Application [6]: Call YQL with JavaScript to get an RSS news feed. Creating YQL Open Data Tables [9]: Create a YQL Open Data Table for the Bay Area Regional Transit (BART) ETA feed. Executable JavaScript in Open Data Tables [12]: Run JavaScript within an Open Data Table to make requests to Web services and perform authorization.

Yahoo! Developer Network

September 8, 2011

YQL Tutorials and Code Examples

First YQL Application


Time Required: 10 min.

Introduction
This tutorial shows how to create a simple Web application that uses YQL to fetch an RSS feed. The call to YQL is made within an HTML script tag, and the returned JSON response is parsed with JavaScript.

Setting Up the Example


1. From your Web server, create a new file called yql_news_app.html. 2. Copy the HTML below into yql_news_app.html. The src attribute in the second script tag is given an empty value intentionally. We will assign a YQL statement to src later. <html> <head><title>YQL and RSS: Yahoo! Top News Stories</title> <style type='text/css'> #results{ width: 40%; margin-left: 30%; border: 1px solid gray; padding: 5px; height: 200px; overflow: auto; } </style> <script type='text/javascript'> // Parses returned response and extracts // the title, links, and text of each news story. function top_stories(o){ var items = o.query.results.item; var output = ''; var no_items=items.length; for(var i=0;i<no_items;i++){ var title = items[i].title; var link = items[i].link; var desc = items[i].description; output += "<h3><a href='" + link + "'>"+title+"</a></h3>" + desc + "<hr/>"; } // Place news stories in div tag document.getElementById('results').innerHTML = output; } </script> </head> <body> <!-- Div tag for stories results --> <div id='results'></div> <!-- The YQL statment will be assigned to src. --> <script src=''></script> </body> </html>

Yahoo! Developer Network

September 8, 2011

YQL Tutorials and Code Examples 3. Run the example query get rss feed from yahoo top stories1 in the YQL Console. Select the JSON radio button and click TEST. You should see the returned JSON response on the Formatted View tab. 4. In the YQL statement, replace the word "title" with an asterisk "*". Click TEST. The asterisk will return all of the fields (rows) of the response. 5. From the returned JSON response, find the results object. Notice that within the results object, there is the array item containing objects with the title, link, and description of each news article. 6. Click Copy URL. From yql_news_app.html, paste the URL into the src attribute of the second script tag as seen below: <body> <div id='results'></div> <script sc'tp/qeyyhopscmv/ulcylqslc%0%0rm2rs2wee2ul3%2tp3%F2rsnw.ao.o%Fs%Fosois2&omtjo&alakcfn' r=ht:/ur.aoai.o/1pbi/q?=eet2*2fo%0s%0hr%0r%D2ht%A2%Fs.esyhocm2rs2tptre%2fra=snclbc=buc> </script> </body> Notice that the YQL statement has been URL-encoded and the format parameter specifies that the response be in JSON. 7. At the end of the URL, replace the callback value 'cbfunc' with 'top_stories'. The new callback function top_stories will be called as soon as YQL returns the response and have access to the returned JSON. 8. Point your browser to yql_news_app.html. Your YQL News Application should resemble the figure below.

Figure 2.1. YQL News Application

9. (Optional) Use almost the exact same code to create an application for finding sushi restaurants in San Francisco. Replace the YQL statement for the Yahoo! Top Stories in your YQL News Application with the example query find sushi restaurants in san francisco2. Be sure that the format is JSON and that the name of the callback function is correct.
1

http://developer.yahoo.com/yql/console/#h=select%20title%20from%20rss%20where%20url%3D%22http%3A//rss.news.yahoo.com/rss/topstories%22 2 http://developer.yahoo.com/yql/console/#h=select%20*%20from%20local.search%20where%20query%3D%22sushi%22%20and%20location%3D%22san%20francisco%2C%20ca%22

Yahoo! Developer Network

September 8, 2011

YQL Tutorials and Code Examples

10. Make a few changes below to the function top_stories as seen below: function top_stories(o){ var items = o.query.results.Result; var output = ''; var no_items=items.length; for(var i=0;i<no_items;i++){ var title = items[i].Title; var link = items[i].Url; var desc = items[i].Rating.LastReviewIntro; output += "<h3><a href='" + link + "'>"+title+"</a></h3>" + desc + "<hr/>"; } document.getElementById('results').innerHTML = output; } 11. The YQL News Application is now the YQL Sushi Restaurant Finder Application.

Figure 2.2. YQL Sushi Restaurant Finder Application

Source Code
YQL Top News Application
<xi:include></xi:include> <html> <head><title>YQL and RSS: Yahoo! Top News Stories</title> <style type='text/css'> #results{ width: 40%; margin-left: 30%; border: 1px solid gray; padding: 5px; height: 200px; overflow: auto; } </style> <script type='text/javascript'> function top_stories(o){ var items = o.query.results.item; var output = ''; var no_items=items.length; for(var i=0;i<no_items;i++){ var title = items[i].title; var link = items[i].link; var desc = items[i].description; output += "<h3><a href='" + link + "'>"+title+"</a></h3>" +

Yahoo! Developer Network

September 8, 2011

YQL Tutorials and Code Examples

desc + "<hr/>"; } document.getElementById('results').innerHTML = output; } </script> </head> <body> <div id='results'></div> <script sc'tp/qeyyhopscmv/ulcylqslc%0%0rm2rs2wee2ul3%2tp3%F2rsnw.ao.o%Fs%Fosois2&omtjo&igotc=as&alaktpsois>/cit r=ht:/ur.aoai.o/1pbi/q?=eet2*2fo%0s%0hr%0r%D2ht%A2%Fs.esyhocm2rs2tptre%2fra=sndansisfleclbc=o_tre'<srp> </body> </html>

Creating YQL Open Data Tables


Time Required: 15 min.

Introduction
This tutorial shows you how create a YQL Open Data Table for the Bay Area Region Transit (BART) Real Time ETA feed. You will then use the YQL console to run a YQL query using the Open Data Table that you created.

What Are Open Data Tables?


YQL already contains an extensive list of built-in tables for you to use that cover a wide range of Yahoo! Web services and access to off-network data. Developers can now add to this list of built-in tables by creating their own Open Data Tables. YQL allows you to create and use your own table definitions, enabling YQL to bind to any data source through the SQL-like syntax and fetch data. Once an Open Data Table is created, anyone can use these definitions in YQL. By creating an Open Data Table for the feed, you can then use YQL to make queries on the data from the feed and mash up other data from existing YQL tables with the social data. For more information about Open Data Tables, see Using YQL Open Data Tables3.

Creating and Using Your Open Data Table


1. Copy bart.xml4 to your Web server. The file bart.xml defines the Open Data Table, which allows the YQL Web service to access data from the BART ETA feed. For a more detailed explanation, see A Closer Look at the Code [10]. 2. Go to the YQL console5. 3. From the YQL console, type the YQL statement below in the "Your YQL Statement" text area and click the button TEST. Be sure to replace "your_domain_name" with the domain name of your Web site. USE "http://your_domain_name/bart.xml" AS bart_table; SELECT * FROM bart_table;

3 4

../../yql/guide/yql-opentables-chapter.html examples/bart.xml.txt 5 http://developer.yahoo.com/yql/console/

Yahoo! Developer Network

September 8, 2011

YQL Tutorials and Code Examples

4. From the "Formatted View" tab, you should see results similar to those below: <results> <station> <name>12th St. Oakland City Center</name> <abbr>12TH</abbr> <date>06/01/2009</date> <time>11:46:00 AM PDT</time> <eta> <destination>Fremont</destination> <estimate>14 min, 29 min</estimate> </eta> <eta> <destination>Millbrae</destination> <estimate>5 min, 20 min, 35 min</estimate> </eta> <eta> <destination>Pittsburg/Bay Point</destination> <estimate>7 min, 22 min, 37 min</estimate> </eta> <eta> <destination>Richmond</destination> <estimate>2 min, 7 min, 17 min</estimate> </eta> <eta> <destination>SF Airport</destination> <estimate>12 min, 27 min, 42 min</estimate> </eta> </station> ... </results> 5. Try a few more YQL statements using the Open Data Tables. Below are a few sample YQL statements. This statement will get you the station name and ETAs for any train going to the San Francisco International Airport. USE "http://your_domain_name/bart.xml" AS bart_table; SELECT name, eta FROM bart_table WHERE eta.destination LIKE "%SF%" This statement gets the station name, the destination of the train, and the ETA to different destinations from the 24th and Mission Street station. USE "http://your_domain_name/bart.xml" AS bart_table; SELECT name, eta.destination, eta.estimate FROM bart_table WHERE name LIKE "%24%"

A Closer Look at the Code


This section will examine in greater detail at the Open Data Table and the YQL queries that are used in this tutorial.

Yahoo! Developer Network

10

September 8, 2011

YQL Tutorials and Code Examples

Open Data Table


The Open Data Table must conform to the XML schema table.xsd6, which is given as the XML namespace attribute for the element table. The element meta has child elements that provide general information about the Open Data Table such as the author, the location of the documentation, and a sample query, as shown below. <?xml version="1.0" encoding="UTF-8"?> <table xmlns="http://query.yahooapis.com/v1/schema/table.xsd"> <meta> <author>Mike Oppentebles</author> <documentationURL>http://yourbartfeed.com/docs</documentationURL> <description>Uses the BART ETA feed.</description> <sampleQuery>SELECT * FROM {table} WHERE abbr="12th"</sampleQuery> </meta> The element bindings allows YQL to map the data source so that information can be queried. The element select defines what repeating element of the XML from the data source will act as a row of a table with the attribute itemPath, which is <root><station>...</station> (given as root.element) in the code below. A good example of a repeating element in XML would be the element entry in an Atom 1.0 feed7. <bindings> <select itemPath="root.station" produces="XML"> <urls> <url>http://www.bart.gov/dev/eta/bart_eta.xml</url> </urls> </select> </bindings>

YQL Query
You invoke an Open Data Table with the verbs USE and AS. YQL fetches the Open Data Table defintion pointed to by the verb USE and then the verb AS creates an alias to that definition. USE "http://your_domain_name/bart.xml" AS bart_table; The alias bart_table can now be accessed with a YQL statement. The simple YQL query below returns all the rows from the BART feed. SELECT * FROM bart_table;

Source Code
<?xml version="1.0" encoding="UTF-8"?> <table xmlns="http://query.yahooapis.com/v1/schema/table.xsd"> <meta> <author>You</author> <documentationURL>None</documentationURL> <description>Uses the BART ETA feed.</description> <sampleQuery>SELECT * FROM {table} WHERE abbr="12th"</sampleQuery> </meta>
6 7

http://query.yahooapis.com/v1/schema/table.xsd http://tools.ietf.org/html/rfc4287#section-1.1

Yahoo! Developer Network

11

September 8, 2011

YQL Tutorials and Code Examples

<bindings> <select itemPath="root.station" produces="XML"> <urls> <url>http://www.bart.gov/dev/eta/bart_eta.xml</url> </urls> </select> </bindings> </table>

Executable JavaScript in Open Data Tables


Time Required: 15-20 min.

Introduction
This tutorial shows how to include JavaScript in an Open Data Table definition that is executed by the YQL Web service. The JavaScript in the Open Data Table for this tutorial includes external libraries and uses OAuth to make an authorized call to the Netflix API. With the Netflix Open Data Table, you can then run YQL statements from the YQL console or create an Open Application that searches and displays movies from the Netflix catalog. Including JavaScript in an Open Data Table definition allows you to do the following: Access external libraries Make requests to Web services Use E4X to structure the response in XML Dynamically control the response with conditional logic Perform Web authorization

Prerequisites
Creating YQL Open Data Tables Register to be a Netflix developer8 and get Netflix API keys9 JavaScript XML

Executable JavaScript
JavaScript is placed in the element execute of an Open Data Table definition. When a SELECT statement calls an Open Data Table definition having the element execute, YQL executes the JavaScript on the backend and returns the response generated by the JavaScript. Open Data Table definitions having the element execute must return a response formed by the JavaScript.

8 9

http://developer.netflix.com/member/register http://developer.netflix.com/member/register

Yahoo! Developer Network

12

September 8, 2011

YQL Tutorials and Code Examples

In contrast, Open Data Tables not having the element execute make GET requests to the URI defined in the element url and return the response from that URI resource. We'll now look at a couple of examples of Open Data Tables with and without the element execute to illustrate the difference. Example of Open Data Table Definition without JavaScript The Open Data Table definition below does not include the element execute, so YQL makes a GET request to the URI defined in the element url: http://www.bart.gov/dev/eta/bart_eta.xml <?xml version="1.0" encoding="UTF-8"?> <table xmlns="http://query.yahooapis.com/v1/schema/table.xsd"> <meta> <author>You</author> <documentationURL>None</documentationURL> <description>Uses the BART ETA feed.</description> <sampleQuery>SELECT * FROM {table} WHERE abbr="12th"</sampleQuery> </meta> <bindings> <select itemPath="root.station" produces="XML"> <urls> <url>http://www.bart.gov/dev/eta/bart_eta.xml</url> </urls> </select> </bindings> </table> Example of Open Data Table Definition with JavaScript Because of the presence of the element execute, YQL executes the JavaScript and returns the response assigned to response.object, which will be discussed in Creating the Response [16]. <?xml version="1.0" encoding="UTF-8"?> <table xmlns="http://query.yahooapis.com/v1/schema/table.xsd"> <meta> <sampleQuery>select * from {table} where a='cat' and b='dog';</sampleQuery> </meta> <bindings> <select itemPath="" produces="XML"> <urls> <url>http://fake.url/{a}</url> </urls> <inputs> <key id='a' type='xs:string' paramType='path' required="true" /> <key id='b' type='xs:string' paramType='variable' required="true" /> </inputs> <execute><![CDATA[ // Your javascript goes here. // We will run it on our servers response.object = <item> <url>{request.url}</url> <a>{a}</a>

Yahoo! Developer Network

13

September 8, 2011

YQL Tutorials and Code Examples

<b>{b}</b> </item>; ]]></execute> </select> </bindings> </table>

A Closer Look at the Code


JavaScript in Open Data Tables definitions have access to the three global objects y, request, and response. From these global objects, methods can be called to include external libraries, perform 2-legged authorization , make requests to Web services, run YQL queries, convert data structures, such as XML to JSON, and more. We'll be looking at the methods used in this tutorial that allow you to include external libraries, make requests to Web services, and extract the results from a returned response. To read more about the JavaScript global objects and the available methods, see the JavaScript Objects and Methods Reference10.

Including External Libraries


The global object y includes the method include that allows you to reference external JavaScript libraries from an Open Data Table definition. Because calls to the Netflix API must be authorized with OAuth, you include external libraries to handle the authorization and sign the request as shown in the code below: <execute> y.include("http://oauth.googlecode.com/svn/code/javascript/oauth.js"); y.include("http://oauth.googlecode.com/svn/code/javascript/sha1.js"); ... </execute>

Making Requests
To make a request to a Web service, you use the method get from either the global object request or from an instance of request; the instance is created by calling the method rest from the global object y. Before looking at the code used in the Open Data Table definition for this tutorial, let's look at a couple of simple examples of using both the global object and an instance of request. This example creates an instance of request 11 by passing the URI resource to the method rest. From the instance, the method get can make the request to the Web service. // Create an instance of the request object var request_instance = y.rest("http://some_web_service"); // Make request to web service with HTTP method 'GET' var returned_response = request_instance.get(); You can also call the method get directly from the global object request. The element url holds the URI to the Web service, which is stored in request.url. ... <select itemPath="" produces="XML"> <urls>
10 11

../../yql/guide/yql-javascript-objects.html ../../yql/guide/yql-javascript-objects.html#yql-execute-requestglobalobject

Yahoo! Developer Network

14

September 8, 2011

YQL Tutorials and Code Examples

<url>http://some_web_service_or_feed</url> </urls> <execute><![CDDATA[ // request.url == 'http://some_web_service_or_feed // This is the same as 'y.rest(request.url).get();' var returned_response = request.get(); ... ]]></execute> </select> ... You can also pass in parameters and define the content returned when making a request to a Web service, which is needed to make authorized requests, such as the request to the Netflix API in this tutorial. To get the response from the Netflix API, the content type must be defined as 'application/xml', and the OAuth authorization header must be passed. The code snippet below also uses the objects response and response.object to extract results from the returned response and create the response that YQL returns. We'll examine the object response in Getting and Creating a Response [15]. // The content type is defined and the OAuth header // is passed to get a response from the Netflix API response.object = request.contentType('application/xml').header("Authorization", OAuth.getAuthorizationHeader("netflix.com", message.parameters)).get().response;

Getting and Creating the Response


Extracting Data from the Returned Response To get the data (results) from the returned response, you reference the object response12 . From this earlier code example, you can see the dot operator being used to reference the results of the returned response. // Create an instance of the request object var request_obj = y.rest("http://some_web_service"); // Make request to web service with HTTP method 'GET' // and get the results by referencing 'response' var response_results = request_obj.get().response; We also saw this example of using the response object to get the data from the Netflix Web service. // The object response lets you extract the // results from the returned response response.object = request.contentType('application/xml').header("Authorization",

12

../../yql/guide/yql-javascript-objects.html#yql-execute-responseglobalobject

Yahoo! Developer Network

15

September 8, 2011

YQL Tutorials and Code Examples

OAuth.getAuthorizationHeader("netflix.com", message.parameters)).get().response; Creating the Response When including JavaScript in an Open Data Table, the response from the YQL Web service is determined by the data that is assigned to the object response.object. You can use the returned value from a Web service or create your own with JSON13 or E4X14. In this code snippet, an XML literal is created with E4X and then assigned to response.object. To interpolate the variable into the XML literal, enclose the variable in braces. // Use E4X to create XML literals with // interpolated variable and assign to // response.object var error = "Failed to get response"; response.object = <error><message>{error}</message></error>; The code from the tutorial assigns the returned XML response to response.object as seen below. If there is an error, a JSON object holding error message is returned. try { // get the content from service along with the OAuth header, and return the result back out response.object = request.contentType('application/xml').header("Authorization", OAuth.getAuthorizationHeader("netflix.com", message.parameters)).get().response; } catch(err) { response.object = {'result':'failure', 'error': err}; }

Paging Results
The element paging lets you have more control over the results returned from the YQL query. We'll look in detail at the paging used in the Netflix Open Data Table that is shown below. For more information about paging, see the Open Data Tables Reference15 and Paging Results16. <paging model="offset"> <start id="start_index" default="0" /> <pagesize id="max_results" max="100" /> <total default="20" /> </paging> The value 'offset' for the attribute model states that the YQL query can state an offset from the start of the results. The ability to get an offset from a result set depends on the source of the data, which in the tutorial is the Netflix API. Be sure to verify that your data source allows retrieving data from an offset when you create future Open Data Tables. <paging model="offset">

13 14 15

http://json.org http://en.wikipedia.org/wiki/ECMAScript_for_XML ../../yql/guide/yql-opentables-reference.html#yql-opentables-paging 16 http://developer.yahoo.com/yql/guide/yql-execute-bestpractices.html#yql-execute-paging

Yahoo! Developer Network

16

September 8, 2011

YQL Tutorials and Code Examples

The default offset is set by the attribute default in the element start, which is 0 or no offset in this example. <start id="start_index" default="0" /> The maximum number of results that can be returned by a YQL query on this table is 100, which is defined by the attribute max. <pagesize id="max_results" max="100" /> The attribute defaut in the element total states the default number of results returned for each page. <total default="20" />

Setting Up This Example


To set up the Netflix Open Data Table and run YQL queries: 1. Copy netflix_table.xml17 to your Web server. The file netflix_table.xml defines the Open Data Table, which allows the YQL Web service to access data from the NetFlix API. For a more detailed explanation, see A Closer Look at the Code [10]. 2. If you do not have Netflix API keys, register for a Netflix developer account18 and apply for an API key19. 3. Go to the YQL Console20. Make sure that the URL of the YQL Console begins with https. 4. Select the first 20 results (see Paging Results [16]) for movies that match the key word "Rocky". Be sure to use your Consumer Key and Consumer Secret from your Netflix account21 for the values for ck and cs. USE "http://your_domain/netflix.xml" AS netflix_table; SELECT * FROM netflix_table WHERE term="rocky" AND ck="your_consumer_key" AND cs="your_consumer_secret" 5. In this YQL statement, you ask for the first 10 results for movies, and the results will include the film title, the URL of the box art image, and the average rating. USE "http://yourdomain/netflix.xml" AS netflix_table; SELECT catalog_title.title, catalog_title.box_art.medium, catalog_title.average_rating FROM netflix_table(10) WHERE term="rocky" AND ck="your_consumer_key" AND cs="your_consumer_secret" 6. You can also specify an offset for the returned results. The YQL statements below returns the five titles and ratings for Star Trek movies with average ratings over 3.0. The results are sorted by the rating in descending order, and the offset is five. USE "http://yourdomain/netflix.xml" AS netflix_table; SELECT catalog_title.title.regular, catalog_title.average_rating FROM netflix_table(5,5) WHERE term="star trek" AND ck="your_consumer_key" AND

17 18

examples/netflix_table.xml.txt http://developer.netflix.com/member/register 19 http://developer.netflix.com/apps/register 20 https://developer.yahoo.com/yql/console/ 21 http://developer.netflix.com/apps/mykeys

Yahoo! Developer Network

17

September 8, 2011

YQL Tutorials and Code Examples

cs="your_consumer_secret" AND catalog_title.average_rating > "3.0" | sort(field="catalog_title.average_rating") | reverse() 7. Optional: Create a Open Application that uses the Netflix Open Data Table Copy yql_netflix_app.html22 to your Web server. 8. Edit yql_netflix_appl.html and insert your Netflix keys23 and the location of your Netflix Open Data Table in the code as shown below: // Place your Netflix keys here to authorize your app NETFLIX.keys = { ck: "use_your_consumer_key", cs:"use_your_consumer_secret" } // Make sure this URI points to your Netflix Open Data Table NETFLIX.odt = "http://your_domain/netflix.xml"; 9. From My Projects24, create a new Open Application. 10. In the Application Editor, specify the URL to netflix_app.html in the "Application URL" field. 11. Click "Preview" to see the canvas view. Your application will have a text field for entering a movie title. Enter "Rocky" and click the button "Find Movie" to see results similar to those in the figure below.

22 23

examples/yql_netflix_app.html.txt http://developer.netflix.com/apps/mykeys 24 http://developer.yahoo.com/dashboard

Yahoo! Developer Network

18

September 8, 2011

YQL Tutorials and Code Examples

12. Try modifying the YQL statement in the code to more precisely return the data that you want from the Netflix API. Take a look at the XML response from a catalog/titles request25 to the Netflix API to see the available data and how it is structured.

Source Code
Netflix Open Data Table
<xi:include></xi:include>

Netflix Application
<xi:include></xi:include>

25

http://developer.netflix.com/docs/REST_API_Reference#0_52696

Yahoo! Developer Network

19

September 8, 2011

YQL Tutorials and Code Examples

YQL Code Examples


Making YQL Queries with JavaScript [20]: Get data from the GeoPlanet API using the OpenSocial library. Making YQL Queries with PHP [22]: Get data from the Upcoming API using PHP with cURL. YQL Social Application [24]: Use YQL with the Yahoo! Social SDK for PHP to make authorized calls to the Flickr API and the Yahoo! Social APIs. YQL INSERT: WordPress Open Application [28]: Post to a WordPress blog from an Open Application using YQL. Getting Updates with YQL [31]: Get Yahoo! Updates from connections with YQL. Data Scraping with YQL [33] Use YQL with XPath to scrape data from Yahoo! Finance.

Prerequisites
YQL First Application [6] Getting Started: Build Your First Open Application26

Making YQL Queries with JavaScript


Summary
This code example shows you how to make YQL queries with JavaScript and the OpenSocial function makeRequest. The YQL query in the code example will get data from the GeoPlanet API based on the user's input.

OpenSocial Method: makeRequest


By using the OpenSocial method gadgets.io.makeRequest27, you can make calls to a Web service with JavaScript without using a crossdomain.xml file. The calls in this example use 2-legged OAuth authorization. Before calling the function makeRequest, define the base URI of the YQL Web service, the YQL query, and a callback function to handle the response, as shown below:

var BASE_URI = 'http://query.yahooapis.com/v1/yql';

// Base URI for Web service var yql_base_uri = "http://query.yahooapis.com/v1/yql"; // Create a variable to make results available // in the global namespace var yql_results = "";
26 27

../../yap/guide/creating_open_app.html http://code.google.com/apis/opensocial/docs/0.7/reference/gadgets.io.html#makeRequest

Yahoo! Developer Network

20

September 8, 2011

YQL Tutorials and Code Examples

// Create a YQL query to get geo data for the // San Francisco International Airport var yql_query = "SELECT * from geo.places WHERE text='SFO'"; // Callback function for handling response data function handler(rsp) { if(rsp.data){ yql_results = rsp.data; } } To make a call to the YQL Web service, you create a URL-encoded query string with the YQL query and the requested format type of the response and append this query string to the base URI of the YQL Web service. The code snippet below shows the base URI with the appended query string for the YQL Web service. Note that the spaces have been URL-encoded. ht:/ur.aoai.o/1pbi/q?=EET2*2FO%0e.lcs2WEE2tx%D2SO2&omtjo tp/qeyyhopscmv/ulcylqSLC%0%0RM2gopae%0HR%0et3%2F%2fra=sn This code example has the JavaScript function toQueryString that creates the query string for the call to the YQL Web service as seen below: // This utility function creates the query string // to be appended to the base URI of the YQL Web // service. function toQueryString(obj) { var parts = []; for(var each in obj) if (obj.hasOwnProperty(each)) { parts.push(encodeURIComponent(each) + '=' + encodeURIComponent(obj[each])); } return parts.join('&'); }; By using a closure in the code below, the variable query, which holds the YQL query, is available to makeRequest. The function toQueryString then builds the query string, which is appended to the base URI by makeRequest before making the call to the YQL Web service.

// Store the anonymous function that wraps // the OpenSocial function makeRequest var runQuery = function(ws_base_uri,query, handler) { gadgets.io.makeRequest(ws_base_uri, handler, { METHOD: 'POST', POST_DATA: toQueryString({q: query, format: 'json'}), CONTENT_TYPE: 'JSON', AUTHORIZATION: 'OAuth' }); }; In the code snippet, the function object runQuery is passed the parameters for makeRequest to call the YQL Web service. The handler returns the response data (JSON object) to the OpenSocial function stringify to be converted to a string so it can be displayed in a div tag.

Yahoo! Developer Network

21

September 8, 2011

YQL Tutorials and Code Examples

<div id="results"></div> <script> // Call YQL Web service and use YQL query // to get results from the GEO runQuery(yql_base_uri,yql_query,handler); // Use stringify function from OpenSocial library // to convert JSON to string and display the string // in the div with the id 'results' document.getElementById('results').innerHTML = gadgets.json.stringify(yql_results.data); </script>

Source Code
<xi:include></xi:include>

Making YQL Queries with PHP


Summary
This example, yql_php.php28, is a simple application that uses cURL to make YQL calls to the Upcoming API. To understand the following material, you should already be familiar with the topics covered in the TwoMinute Tutorial [1].

Building the YQL URL


To build the YQL URL, append the YQL query to the base URL of the YQL Web service. 1. First, assign the base URL of the YQL Web service to a variable. $yql_base_url = "http://query.yahooapis.com/v1/public/yql"; 2. Now create the YQL query to the Upcoming API and append it to the YQL base URL. YQL queries are passed as query strings, so they must be URL encoded. The query string must begin with q=. $yql_query = "select * from upcoming.events where location='San Francisco' and search_text='dance'"; $yql_query_url = $yql_base_url . "?q=" . urlencode($yql_query); 3. The YQL Web service returns XML as the default format. This code example requests JSON by appending the name-value pair format=json to the query string as shown. $yql_query_url .= "&format=json";

28

./examples/yql_php.phps

Yahoo! Developer Network

22

September 8, 2011

YQL Tutorials and Code Examples 4. Run the query29 in the YQL console and look at both the URL in The REST query window and the JSON response in the Formatted View window.

Calling the YQL Web Service with cURL


Calling the YQL Web service with cURL only requires three lines of code. After initializing the call by passing the curl the YQL URL, you request a response with curl_setopt and then execute the call. $session = curl_init($yql_query_url); curl_setopt($session, CURLOPT_RETURNTRANSFER,true); $json = curl_exec($session); To make the response easier to handle, convert the JSON response to a PHP object to easily access data. $phpObj = json_decode($json);

Parsing the Response


The YQL Web service will always return results wrapped in the query field with meta data. If the YQL query returns data, the data is wrapped in the results field, which you can see by running this query30. If no data is returned, the results element in the XML response is empty or the results field in JSON is null. Therefore, before parsing the response returned by YQL, your code should always check the results field as shown below: if(!is_null($phpObj->query->results)){ // Safe to parse data } The structure of the data within the results field that is returned by YQL is different for each API. The repeated field (like a row in an SQL table) in the returned response in this code example is the event field. The code below extracts and displays data from each event: if(!is_null($phpObj->query->results)){ foreach($phpObj->query->results->event as $event){ $events .= "<div><h2>" . $event->name . "</h2><p>"; $events .= html_entity_decode(wordwrap($event->description, 80, "<br/>")); $events .="</p><br/>$event->venue_name<br/>$event->venue_address<br/>"; $events .="$event->venue_city, $event->venue_state_name"; $events .="<p><a href=$event->ticket_url>Buy Tickets</a></p></div>"; } }

Source Code
<xi:include></xi:include>

29

http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20upcoming.events%20where%20location%3D%22San%20Francisco%22%20and%20search_text%3D%22dance%22&format=json 30 http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20upcoming.events%20where%20location%3D%22San%20Francisco%22%20and%20search_text%3D%22dance%22&format=json

Yahoo! Developer Network

23

September 8, 2011

YQL Tutorials and Code Examples

YQL Social Application


Summary
This example, yql_basic.php31, is a simple application that uses the Yahoo! Social SDK for PHP to make YQL calls. Using YQL with the SDK is ideal because the SDK will handle your OAuth authorization, which is required to access the Social Directory APIs. YQL, in turn, extends the functionality of the SDK by enabling access to the public data from Yahoo! Web services and external data such as RSS and Atom feeds. To understand the following material, you should already be familiar with the topics covered in My Social PHP Application32 and the Two-Minute Tutorial [1].

Initial Code
Before proceeding with the code that makes YQL queries, be sure to include the PHP SDK and define an API Key and a Shared Secret:

// Include the PHP SDK. include_once("yosdk/lib/Yahoo.inc"); // Define constants to store your API Key (Consumer Key) and // Shared Secret (Consumer Secret). define("API_KEY","your_Consumer_Key_goes_here"); define("SHARED_SECRET","your_Shared_Secret_goes_here");

Querying Public Data


There are great number of sources for public data on the Web that can be accessed with YQL. Although public, many of these sources require authorization. With the API Key and the Shared Secret provided by Yahoo!, you can use the PHP SDK to perform two-legged authorization with the YahooApplication class. (If you need to read up on two-legged authorization, see Private Data v. Public Data33.) From a YahooApplication object, invoke the method query, which calls the YQL Web service (that is, runs the YQL queries). The following code snippets show you how to perform two-legged authorization with YahooApplication and how to make YQL queries. The API Key and Shared Secret are passed to the constructor of the YahooApplication class. If the application has been authorized by Yahoo!, the YahooApplication object $two_legged_app is returned:

$two_legged_app = new YahooApplication(API_KEY,SHARED_SECRET); Next, the application defines two YQL queries: one to the Flickr API, another to the Yahoo! News RSS feed. You can also run these queries in the YQL Console34:

$flickr_query =
31 32 33

./examples/yql_basic.phps ../../yap/guide/other-code-exs.html#my_social ../../oauth/guide/about.html#oauth-private_public_data 34 http://developer.yahoo.com/yql/console/

Yahoo! Developer Network

24

September 8, 2011

YQL Tutorials and Code Examples

"select * from flickr.photos.search where text=\"panda\" limit 3"; $news_feed = "select * from rss where url='http://rss.news.yahoo.com/rss/topstories' and title LIKE \"%China%\""; With the $two_legged_app object, the application calls the query method. The var_dump shows the structure and contents of the query response of the YQL Web service:

$flickrResponse = $two_legged_app->query($flickr_query); var_dump($flickrResponse); $newsResponse = $two_legged_app->query($news_feed); var_dump($newsResponse);

Querying Private Data


Unlike YQL queries for public data, you need user authorization to obtain data from the Social Directory APIs (Profiles, Updates, Connection, Contacts, and Presence). During the process of user authorization, the user will be redirected to the Yahoo! login page and then asked to authorize your application. You initiate this three-legged authorization process with the YahooSession class. (If you are unfamiliar with the term "three-legged authorization," you may want to read Private Data v. Public Data35.) Getting a valid YahooSession object means that the user has authorized your application, allowing you to access this user's private data. The following code snippets show how to perform authorization with YahooSession and how to use YQL to access private social data from Yahoo! The method requireSession makes sure the user has logged into Yahoo!, redirecting to the Yahoo! login page if necessary. A YahooSession object is returned, confirming that the OAuth authorization has been completed:

$session=YahooSession::requireSession(API_KEY, SHARED_SECRET); The YQL queries and API names are assigned to the associative array $api_queries:

// Define YQL queries for the Social Directory APIs $profile = "select * from social.profile where guid=me"; $contacts = "select fields.value from social.contacts where guid=me"; $connections = "select * from social.connections where owner_guid=me"; $updates = "select * from social.updates where guid=me"; $status = "select value.status from social.presence where guid=me"; $api_queries = array("Profiles"=>$profile, "Contacts"=>$contacts, "Connections"=>$connections, "Updates"=>$updates, "Presence"=>$status); The YQL query is made from the YahooSession object $session with the method query. The use of var_dump allows you to see the contents of the returned response for each YQL query:

35

../../oauth/guide/about.html#oauth-private_public_data

Yahoo! Developer Network

25

September 8, 2011

YQL Tutorials and Code Examples

// Make the calls to YQL and dump the responses. foreach($api_queries as $api=>$query) { echo "<h2>$api Data</h2>"; $queryResponse = $session->query($query); if ($queryResponse == NULL) { echo "<p>"; echo "Error: No query response for $api."; echo " Check your permissions. Also, check the syntax of the YQL query."; echo "</p>"; } else { echo "<pre>"; var_dump($queryResponse); echo "</pre>"; }

The following listing shows the var_dump from the YQL query made on the Profiles API.

Profiles Data array(1) { ["query"]=> array(7) { ["count"]=> string(1) "1" ["created"]=> string(20) "2008-10-08T06:11:21Z" ["lang"]=> string(5) "en-US" ["updated"]=> string(20) "2008-10-08T06:11:21Z" ["uri"]=> string(80) "http://query.yahooapis.com/v1/yql?q=select+*+from+social.profile+where+guid%3Dme" ["diagnostics"]=> array(4) { ["url"]=> array(2) { ["execution-time"]=> string(1) "9" ["content"]=> string(89) "http://social.yahooapis.com/v1/users.guid(UQIDZJNWVNLQD4GXZ5NGMZUSTQ4)/profile" } ["user-time"]=> string(2) "11"

Yahoo! Developer Network

26

September 8, 2011

YQL Tutorials and Code Examples

["service-time"]=> string(1) "9" ["build-version"]=> string(16) "2008.10.06.06:05" } ["results"]=> array(1) { ["profile"]=> array(16) { ["xmlns"]=> string(41) "http://social.yahooapis.com/v1/schema.rng" ["yahoo"]=> string(36) "http://www.yahooapis.com/v1/base.rng" ["uri"]=> string(70) "http://social.yahooapis.com/v1/user/UQIDWJJWVNQD4GXZ5NGMZUSTQ4/profile" ["guid"]=> string(26) "UQIDWJNWVNQD4GXZ5NGMZUSTQ4" ["created"]=> string(20) "2008-09-16T05:06:29Z" ["familyName"]=> string(2) "Me" ["gender"]=> string(1) "M" ["givenName"]=> string(2) "Me" ["image"]=> array(4) { ["height"]=> string(3) "192" ["imageUrl"]=> string(102) "tp/F.aof.o/oed41bcb1eols1tdL0clV6.tEY//12jgcAxJ_vUD" ht:/3yhoscmcri/0ec8i2zu3p/COzr6u9s2NS-1t9.p?iIOBGKNs ["size"]=> string(7) "192x192" ["width"]=> string(3) "192" } ["interests"]=> array(2) { ["declaredInterests"]=> string(13) "Cloud bathing" ["interestCategory"]=> string(13) "prfFavHobbies" } ["lang"]=> string(5) "en-US" ["location"]=> string(17) "San Francisco, CA" ["nickname"]=> string(11) "blisterHead" ["profileUrl"]=>

Yahoo! Developer Network

27

September 8, 2011

YQL Tutorials and Code Examples

string(54) "http://profiles.yahoo.com/u/UQIDWJNWIQ7NQD4GXZ5NGMZUSTQ4" ["timeZone"]=> string(19) "America/Los_Angeles" ["isConnected"]=> string(4) "true" } } } }

Source Code
<xi:include></xi:include>

YQL INSERT: WordPress Open Application


Summary
This code example shows how to use the YQL INSERT statement in an Open Application to post to your WordPress blog. YML is used to create a simple HTML form for the UI, and PHP is used to call the YQL Web service to run the YQL INSERT statement.

Prerequisites
YQL INSERT and DELETE Statements HTML Form in a Canvas View36 Getting Started: Build Your First Open Application37 Install Yahoo! Social SDK for PHP38 Host a WordPress blog39 or register for a WordPress.com account40

Small View: yml:form


The WordPress Open Application uses the YML tag yml:form for entering information. The attribute params is like the HTML form attribute action. In the code snippet below, the user triggers the call to the script yql_insert_wordpress.php when submitting the form by clicking "Publish".

<yml:form name="wordpress" params="yql_insert_wordpress.php" method="POST" insert="blog_sect"> <fieldset> <b>Username:</b> <input type="text" name="username"/><br/> <b>Password:</b> <input type="password" name="password" /><br/>
36 37

http://developer.yahoo.com/yap/guide/form.html ../../yap/guide/creating_open_app.html 38 ../../social/sdk/php 39 http://wordpress.org/download/ 40 http://en.wordpress.com/signup/

Yahoo! Developer Network

28

September 8, 2011

YQL Tutorials and Code Examples

<b>Blog URL:</b> <input type="text" name="blogurl" /><br/> <input type="hidden" name="blog_form"/> <p><b>Title:</b> <input type="text" name="title"/></p> <p><b>Add New Post:</b><br/> <textarea name="blog" rows='5' cols='60'>Enter your blog post here.</textarea><br/></p> <input type="submit" name="submit" value="Publish" /> </fieldset> </yml:form> The attribute insert places the returned response from yql_insert_wordpress.php into the div tag "blog_sect". If "Publish" is clicked again, the contents in this div will be replaced with the new content returned by yql_insert_wordpress.php.

<yml:form name="wordpress" params="yql_insert_wordpress.php" method="POST" insert="blog_sect"> ... </yml:form> <div id="blog_sect"></div>

YQL: INSERT
The syntax for the YQL INSERT statement follows that of the SQL INSERT statement: INSERT INTO (table) (list of comma separated field names) VALUES (list of comma separated values) Try running the following YQL INSERT statement41 that references the WordPress Open Table and then view the results at http://yqlblog.wordpress.com 42. USE 'http://www.datatables.org/wordpress/wordpress.post.xml' AS wordpress.post; INSERT INTO wordpress.post(title, description, blogurl, username, password) VALUES ("YQL meets WordPress", "Posting with YQL", "http://yqlblog.wordpress.com", "yqlblog", "password")

Calling the YQL Web Service


Building the URL
The URL for making a request to the YQL Web service has a base URL and a query string. The query string contains the YQL statement, the requested response format, and any environment files. The environment file at http://datatables.org/alltables.env includes the USE statement for the WordPress Table, which creates the alias wordpress.post as seen below: use 'http://www.datatables.org/wordpress/wordpress.post.xml' as wordpress.post; Below, the URL used to call the YQL Web service is divided and stored in variables to illustrate the different components: $yql_base_url holds the base URL to the YQL Web service, $yql_insert holds the
41

http://developer.yahoo.com/yql/console/?q=insert%20into%20wordpress.post%20(title%2C%20description%2C%20blogurl%2C%20username%2C%20password)%20values%20(%22YQL%20meets%20WordPress%22%2C%20%22Posting%20with%20YQL%22%2C%20%22http%3A%2F%2Fyqlblog.wordpress.com%22%2C%20%22yqlblog%22%2C%20%22password%22)&env=http://datatables.org/alltables.env 42 http://yqlblog.wordpress.com

Yahoo! Developer Network

29

September 8, 2011

YQL Tutorials and Code Examples

YQL INSERT statement, and $yql_format and $yql_env_tables hold the request format and URL to the environment file respectively. // Base URL to the YQL Web service $yql_base_url = 'https://query.yahooapis.com/v1/public/yql'; // YQL Insert Statement $yql_insert = "INSERT INTO wordpress.post(title,description,blogurl,username,password) " . "VALUES(\"$title\",\"$blog\",\"$blogurl\",\"$username\",\"$password\")"; // Request XML as the response format $yql_format = "&format=xml&env="; // Include environment files that references // WordPress Open Data Table $yql_env="http://datatables.org/alltables.env"; Because the YQL INSERT statement is sent via POST with cURL, we build the POST fields by concatenating the various components. We configure the cURL call to send the POST fields in Calling the YQL Web Service [30]. Notice that in this code snippet only $yql_format is not URL-encoded because it contains the characters '&' and '='.

$post_fields = "q=" . rawurlencode($yql_insert) . $yql_format . rawurlencode($yql_env));

Sending the Request to YQL


Before making the request with cURL, let's look again at the variables that hold the base URL and the post fields from Building the URL [29]: // Base URL to the YQL Web service $yql_base_url = 'https://query.yahooapis.com/v1/public/yql'; $post_fields = "q=" . rawurlencode($yql_insert) . $yql_format . rawurlencode($yql_env)); The cURL call for this code example is fairly typical. You need to configure cURL to send the request via POST and accept POST fields, ask for a returned response, and turn off SSL verification for both host and peer certificates: // Initialize the curl call with the URI to the YQL Web service $ch = curl_init($yql_url); // Set the request to include POST fields // URL encode the INSERT statement and the // URL to the environment curl_setopt($ch, CURLOPT_POST, 1); curl_setopt($ch, CURLOPT_POSTFIELDS, $post_fields); // Configure the request to get response // Turn off SSL verification of peer and host certification curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);

Yahoo! Developer Network

30

September 8, 2011

YQL Tutorials and Code Examples

curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false); // Execute the curl call and parse results $x = curl_exec($ch); $s = simplexml_load_string($x);

Source Code
Small View Code
<!-- Creates user badge --> <yml:user-badge uid="viewer" linked="true" width="55" reflexive="false" useyou="true" capitalize="true"/> <!-- Form to enter and post blog --> <yml:form name="wordpress" params="yql_insert_wordpress.php" method="POST" insert="blog_sect"> <fieldset> <b>Username:</b> <input type="text" name="username"/><br/> <b>Password:</b> <input type="password" name="password" /><br/> <b>Blog URL:</b> <input type="text" name="blogurl" /> <br/> <input type="hidden" name="blog_form"/> <p><b>Title:</b> <input type="text" name="title"/></p> <p><b>Add New Post:</b><br/> <!-- Text area for entering blog --> <textarea name="blog" rows='5' cols='60'>Enter your blog post here.</textarea><br/></p> <input type="submit" name="submit" value="Publish" /> </fieldset> </yml:form> <!-- Div element that holds returned response from "yql_insert_wordpress.php" --> <div id="blog_sect"></div>

WordPress Open Application


<xi:include></xi:include>

Getting Updates with YQL


Summary
This code example shows you how to use YQL with the Yahoo! Social SDK for PHP to get user updates and the updates for the user's connections. The SDK handles the OAuth authorization, and YQL fetches the user data. To understand the following material, you should already be familiar with the topics covered in My Social PHP Application43 and the Two-Minute Tutorial [1].

43

../../yap/guide/other-code-exs.html#my_social

Yahoo! Developer Network

31

September 8, 2011

YQL Tutorials and Code Examples

Prerequisites
Getting Started: Build Your First Open Application44 MySocial PHP Application45

YQL Queries
The syntax of YQL is similar to that of SQL, and the YQL Web service, like a MySQL database, returns data based on queries. The YQL Web service, however, returns the data in the form of XML or JSON. For more information about making YQL queries, see SELECT Statement [6]. YQL has a special literal me that holds the GUID of the currently logged in Yahoo! user. This YQL query uses this special literal to get the data of five connections of the currently logged in Yahoo! user. SELECT * FROM social.connections WHERE owner_guid=me LIMIT 5 From the YQL response for the query above, you can then extract the GUID for each connection. These GUIDs can then be matched against the key guid in a new YQL query to get the updates for each connection as shown in the code snippet below: SELECT * FROM social.updates WHERE guid='GUID_of_a_connection' LIMIT 5 Now that you have some familiarity with YQL and its syntax, try using the YQL Console46 to experiment with different YQL queries and different tables such as Flickr and Search.

Calling the YQL Web Service with the PHP SDK


The code example uses the method requireSession from the class YahooSession to get authorization to access user data and the method query to call the YQL Web service. The code snippets below will focus on authorization and YQL queries. This one line of code does a lot of work to complete the OAuth authorization. The method requireSession first looks for an existing session, and if it's not found, redirects the user to the Yahoo! login page to sign in. The user can then authorize the app to access private data. The return value of requireSession is a YahooSession object (a session), which you will use next to make YQL queries.

// Authorize the app to get user data $session = YahooSession::requireSession($consumerKey, $consumerKeySecret);

To call the YQL Web service, pass the YQL query to the method query from the YahooSession object $session. In the code snippet below, the results of the queries are passed to the variables $user_updates and $user_connections to be parsed for specific data.

// Use the SDK method 'query' from the YahooSession class // Limit the results to 5 updates and 5 connections $user_updates = $session->query("SELECT * FROM social.updates WHERE
44 45

../../yap/guide/creating_open_app.html ../../yap/guide/other-code-exs.html#my_social 46 http://developer.yahoo.com/yql/console/

Yahoo! Developer Network

32

September 8, 2011

YQL Tutorials and Code Examples

guid=me LIMIT 5"); $user_connections = $session->query("SELECT * FROM social.connections WHERE owner_guid=me LIMIT 5");

Source Code
<xi:include></xi:include>

Data Scraping with YQL


Summary
This example, yql_html_scraper.html47, uses YQL to scrape HTML from Yahoo! Finance and then creates customized output. The YQL module for YUI48 is used to call the YQL Web service.

Using the YQL Module for YUI


You can can download the YQL module for YUI from GitHub49 or include it in your Web page like any other JavaScript file. The steps below will show you how to use the YQL module to call the YQL Web service. 1. Include the YUI library and the YQL module in your Web page. <script type="text/javascript" src="http://yui.yahooapis.com/3.0.0/build/yui/yui-min.js"></script> <script type="text/javascript" src="http://yui.yahooapis.com/gallery-2010.01.27-20/build/gallery-yql/gallery-yql-min.js"></script> 2. Create a YUI instance50 to load the Node and YQL modules. YUI().use('gallery-yql', 'node', function(Y) { // The code for calling the YQL Web service and parsing the response goes here. }) 3. Using the html table, create a YQL query to get data from Yahoo! Finance. The variable ticker stores the stock symbol of the company given by the user. See Using XPath in YQL Statements [34] for details about the YQL query. var yql_query = 'select * from html where url="http://finance.yahoo.com/q?s=' + ticker + '"' + " and xpath=" + "'//div[@id=" + '"yfi_headlines"' + "]/div[2]/ul/li/a'"; 4. Instantiate a YQL object by passing the YQL query to the constructor. var yql_obj = new Y.yql(yql_query);

47 48 49

./examples/yql_yui_html_scraper.html.txt http://davglass.github.com/yui-yql/ http://github.com/davglass/yui-yql 50 http://developer.yahoo.com/yui/3/examples/yui/yui-core.html

Yahoo! Developer Network

33

September 8, 2011

YQL Tutorials and Code Examples

5. To initiate the call to the YQL Web service, call the on method from the YQL object. yql_obj.on('query', function(response) { // Handle and parse the response here }) 6. Make sure that results were returned and then parse those results. if(response.results){ var no_stories = response.results.a.length; var headlines = "<p>"; for(var i=0;i<no_stories;i++){ headlines+= "<a href='" + response.results.a[i].href + "'>" + response.results.a[i].content.split('\n').join('').replace(/\s+/gi, " ") + "</a><br/>"; } headlines += "</p>"; document.getElementById("results").innerHTML = headlines; }else{ headlines += "Sorry, could not find any headlines for the ticker symbol " + ticker + ". Please try another one."; } Because each a element is a seen as row of data by YQL, you need to iterate through each a element to access the href and content child elements. Run the query51 in the YQL Console to examine the data structure or the returned response in greater detail.

Using XPath in YQL Statements


The YQL statement below is used in this code example to access the links for headline stories for the company represented by {ticker_sym}. This section will examine the data source for the YQL statement and the XPath expression that extracts the links. select * from html where url="http://finance.yahoo.com/q?s={ticker_sym}" and xpath='//div[@id="yfi_headlines"]/div[2]/ul/li/a' The YQL query in this example allows you to access HTML using the html table. YQL also has tables for accessing other data formats, such as XML, CSV, JSON, RSS, XST, etc. select * from html The key url determines the data source for the YQL query. The data source in this example is Yahoo! Inc (YHOO) page on Yahoo! Finance. url="http://finance.yahoo.com/q?s=yhoo" The XPath statement below extracts the bulleted links from the third div child of the div with the id of "yfi_headlines". Each link (a element) is returned as a row (repeated element in XML). xpath='//div[@id="yfi_headlines"]/div[2]/ul/li/a' For more information about XPath, see WC3 XPath Language52.
51 52

http://y.ahoo.it/ogX/hpXo http://www.w3.org/TR/xpath/

Yahoo! Developer Network

34

September 8, 2011

YQL Tutorials and Code Examples

Source Code
<xi:include></xi:include>

Sample Open Data Tables


The following table has links to examples of YQL Open Data Tables: Sample Code Flickr Photo Search [14] Access to Digg Events using Gnip [15] Hello World Table [51] Yahoo! Messenger Status [51] OAuth Signed Request to Netflix [52] Request for a Flickr frob [54] Description Allows you to retrieve data from a Flickr photo search. Allows you to retrieve Digg activities using the Gnip API. Allows you to search a fictional table in which "a" is the path and "b" is the term. Allows you to see the status of a Yahoo! Messenger user. Allows you to make a two-legged OAuth signed request to Netflix. Returns the Flickr frob, which is analogous to the request token in OAuth.

Celebrity Birthday Search using IM- Retrieves information about celebrities whose birthday is today DB [55] by default, or optionally on a specific date. Share Yahoo! Applications [58] Provides a list of Yahoo! Applications that you and your friends have installed, indicating whether each app is installed exclusively by you, your friends, or both. Allows you to filter HTML using CSS selectors. Shortens a long URL into a bit.ly link.

CSS Selector for HTML [60] Bit.ly Shorten URL (INSERT) [24]

YQL Screencasts
A screencast is a multimedia demonstration of an application, a tool, or a service. Screencasts guide you through Yahoo! APIs and related tools using audio and video. The following screencasts are related to YQL: Screencast Introducing YQL
53

Description

Author/Source

This is an introductory screencast to YQL using the Jatin Billimoria, YDN YQL Console.

YDN Screencast: YQL54 Discusses how to access public and private Web Dan Theurer and Jonathan services, filter and join results, extract data from Trevor, YDN web pages, and extend YQL to query additional data sources. YQL Execute55 This is an intermediate-level screencast that dis- Sam Pullara and Nagesh cusses Open Data Tables and the use of JavaScript Susarla, YDN to manipulate data within Open Data Tables.

53 54

http://developer.yahoo.com/yos/screencasts/yql_screencast.html http://developer.yahoo.net/blogs/theater/archives/2009/03/ydn_screencast_yql.html 55 http://developer.yahoo.net/blogs/theater/archives/2009/04/yql_execute_screencast.html

Yahoo! Developer Network

35

September 8, 2011

YQL Tutorials and Code Examples

Screencast

Description

Author/Source

Screencast: Building an This advanced-level screencast shows you how to Christian Heilmann, htonline profile of distrib- use YQL, a YUI CSS grid, a few dozen lines of tp://www.wait-till-i.com uted data with YQL56 PHP, and a bit of CSS to create your own personal online profile. YQL Console Update57 This screencast discusses updates to the YQL Paul Donnelly and Jatin Console, including the REST query box and query Billimoria, YDN aliases. YQL Query Builder58 This screencast discusses the query builder feature, Paul Donnelly and Jatin which allows developers who are new to YQL to Billimoria, YDN explore tables and create queries easily and quickly.

YQL Slideshows
The following slideshows discuss YQL and its usage: Slideshow Description Author/Source

YQL - A Query Language for Jonathan Trevor, YQL Lead, gives an overview Jonathan Trevor, Slidethe Web59 of the YQL Web service and YQL language. share Open Hack London - Introduc- This narrated slideshow introduces YQL and uses Christian tion to YQL60 code example to show its main features. Slideshare Heilmann,

YQL, Flickr, OAuth, YAP61 This slideshow offers a step-by-step explanation Erik Eldridge, Slideshare of how to use YQL and OAuth together to get Flickr photos. BayJax July 2009 - Browser In this slideshow, you learn how to use YQL and Jonathan LeBlanc, SlideMVC with YQL & YUI62 YUI together to create widgets. YQL retrieves share the data (the model layer), and YUI handles the presentation and control layers.

Additional Tutorials and Code Examples


The following advanced tutorials and code examples show specialized uses for YQL: Tutorial/Code Example Description Author/Source

Building a (re)search interface for Uses a single YQL statement with an Chris Heilmann, htYahoo, Bing and Google with Open Data Table to make queries to tp://www.wait-till-i.com YQL63 three search engines. Getting stock information with YQL Turns Yahoo! Finance into an API with YQL Team, http://www.yqland open data tables64 executable JavaScript in an Open Data blog.net/blog Table.
56 57 58

http://www.wait-till-i.com/2009/04/15/screencast-building-an-online-profile-of-distributed-data-with-yql/ http://developer.yahoo.com/blogs/ydn/posts/2010/03/yql_console_changes/ http://developer.yahoo.com/blogs/ydn/posts/2010/10/yql-query-builder-and-explorer/ 59 http://www.slideshare.net/sh1mmer/yql-a-query-language-for-the-web 60 http://www.slideshare.net/cheilmann/open-hack-london-introduction-to-yql 61 http://www.slideshare.net/erikeldridge/yql-flickr-oauth-yap 62 http://www.slideshare.net/jcleblanc/bayjax-july-2009-browser-mvc-with-yql-yui#notesList 63 http://www.wait-till-i.com/2009/12/09/building-a-research-interface-for-yahoo-bing-and-google-with-yql/ 64 http://www.yqlblog.net/blog/2009/06/02/getting-stock-information-with-yql-and-open-data-tables/

Yahoo! Developer Network

36

September 8, 2011

YQL Tutorials and Code Examples

Tutorial/Code Example

Description

Author/Source

Tutorial: scraping and turning a Scrapes a website with YQL and XPath Chris Heilmann, htwebsite into a widget with YQL65 to make a TV fun facts widget. tp://www.wait-till-i.com

Additional References and Resources


The following references and resources for YQL include libraries, handouts, blog posts, and demos: Reference/Resource YQL Module for YUI
66

Description

Author/Source

JavaScript library for making queries to the Dav Glass, http://yuilibYQL Web service. rary.com

Python-YQL: client library for Python library for making queries to the YQL Stuart Colville, http://muffinYQL67 Web service. research.co.uk Using YQL for Developers Handout that briefly introduces YQL and YDN, http://developer.ya(PDF)68 serves as a cheat sheet. hoo.com Creating Open Data Tables for Handout that introduces and provides ex- YDN, http://developer.yaYQL (PDF)69 amples of YQL Open Data Tables. hoo.com YQL: INSERT INTO inter- YDN blog post that discusses the YQL IN- Jonathan Trevor, YDN net70 SERT INTO statement. YQL examples demo links are also given. GooHooBi Search71 YQL-powered application that returns search Chris Heilmann, http://icresults from Google, Yahoo!, and Bing. ant.co.uk YDN

YQL and YUI: Building Video of Chris Heilmann talking about how Chris Heilmann, Blocks for Quick Applica- to use YQL and YUI to simplify Web devel- Theater73 tions72 opment. YQL Console Update - Paul Paul Donnelly from the YQL team discusses Paul Donnelly, YDN Donnelly74 changes to the YQL console including Query Aliasing.

65 66

http://www.wait-till-i.com/2009/08/25/tutorial-scraping-and-turning-a-web-site-into-a-widget-with-yql/ http://yuilibrary.com/gallery/show/yql 67 http://muffinresearch.co.uk/archives/2009/11/24/python-yql-client-library-for-yql/ 68 http://developer.yahoo.com/yql/yql-users-v0.0.2.pdf 69 http://developer.yahoo.com/yql/yql-opendatatables-v0.0.2.pdf 70 http://developer.yahoo.net/blog/archives/2009/07/yql_insert.html 71 http://icant.co.uk/goohoobi/ 73 http://developer.yahoo.com/yui/theater/ 72 http://developer.yahoo.com/yui/theater/video.php?v=heilmann-yql 74 http://developer.yahoo.net/blogs/theater/archives/2010/03/yql_console_changes.html

Yahoo! Developer Network

37

September 8, 2011

Using YQL and Open Data Tables

Using YQL and Open Data Tables


Abstract This part of the YQL Guide focuses on the basics of using YQL and Open Data Tables. It covers SELECT statements and YQL response data.

Table of Contents
1. Overview for YQL Users ................................................................................................... 1 How to Run YQL Statements ......................................................................................... 1 The Two-Minute Tutorial .............................................................................................. 1 YQL Web Service URLs ............................................................................................... 2 YQL Query Parameters ......................................................................................... 3 YQL Query Aliases .............................................................................................. 3 Summary of YQL Statements ......................................................................................... 5 Authorization .............................................................................................................. 5 2. SELECT Statement .......................................................................................................... 6 Introduction to SELECT ............................................................................................... 6 Syntax of SELECT ...................................................................................................... 6 Specifying the Elements Returned (Projection) .................................................................. 7 Filtering Query Results (WHERE) .................................................................................. 7 Remote Filters .................................................................................................... 7 Local Filters ....................................................................................................... 8 Combining Filter Expressions (AND, OR) ................................................................ 9 Joining Tables With Sub-selects .................................................................................... 10 Paging and Table Limits .............................................................................................. 10 Remote Limits ................................................................................................... 11 Local Limits ..................................................................................................... 11 Sort and Other Functions ............................................................................................. 12 Remote and Local Processing ....................................................................................... 12 Example With Remote and Local Steps .................................................................. 13 Summary of Remote and Local Controls ................................................................ 14 GUIDs, Social, and Me ............................................................................................... 14 Variable Substitution in the GET Query String ................................................................. 14 Extracting HTML Content With XPath ........................................................................... 15 3. Using the Query Builder .................................................................................................. 17 YQL Query Builder Overview ...................................................................................... 17 Query Builder Usage .................................................................................................. 18 Loading a Query ................................................................................................ 18 Entering Values ................................................................................................. 19 4. INSERT, UPDATE, and DELETE (I/U/D) Statements ........................................................... 20 Introduction to INSERT, UPDATE, and DELETE (I/U/D) Statements .................................. 20 Bindings Required for I/U/D ........................................................................................ 20 JavaScript Methods Available to I/U/D ........................................................................... 22 Syntax of I/U/D ......................................................................................................... 22 Syntax of INSERT ............................................................................................. 22 Syntax of UPDATE ............................................................................................ 22 Syntax of DELETE ............................................................................................ 23 Limitations of I/U/D ................................................................................................... 23 Open Data Table Examples of I/U/D .............................................................................. 24 Bit.ly Shorten URL (INSERT) .............................................................................. 24 WordPress Post (INSERT) ................................................................................... 25 5. Response Data ............................................................................................................... 34 Supported Response Formats ....................................................................................... 34 JSONP-X: JSON envelope with XML content ................................................................. 34 Structure of Response ................................................................................................. 36 Information about the YQL Call .................................................................................... 36 XML-to-JSON Transformation ..................................................................................... 37 JSON-to-JSON Transformation .................................................................................... 38

Yahoo! Developer Network

iii

September 8, 2011

Using YQL and Open Data Tables

Errors and HTTP Response Codes ................................................................................. 39 6. Using YQL Open Data Tables ........................................................................................... 41 Overview of Open Data Tables ..................................................................................... 41 Invoking an Open Data Table Definition within YQL ........................................................ 41 Invoking a Single Open Data Table ........................................................................ 41 Invoking Multiple Open Data Tables ...................................................................... 42 Invoking Multiple Open Data Tables as an Environment ............................................ 42 Working with Nested Environment Files ................................................................. 43 Setting Key Values for Open Data Tables ........................................................................ 44 Using SET to Hide Key Values or Data .................................................................. 45

Yahoo! Developer Network

iv

September 8, 2011

List of Figures
3.1. YQL Query Builder ...................................................................................................... 43 6.1. Environment File Transversal ......................................................................................... 43

Yahoo! Developer Network

September 8, 2011

Chapter 1. Overview for YQL Users


In this Chapter: How to Run YQL Statements [1] The Two-Minute Tutorial [1] YQL Web Service URLs [2] Summary of YQL Statements [5] Authorization [5]

How to Run YQL Statements


You can run YQL statements [5] in the following ways: YQL Console: In your browser with the YQL Console1. SELECT statements using HTTP GET: A Web application can use an HTTP GET request when running SELECT statements, specifying the YQL statement as a query parameter of the Web Service URL [2]. INSERT, UPDATE, and DELETE statements using HTTP POST, PUT, or DELETE: A Web application can similarly use an HTTP GET, PUT, or DELETE for INSERT, UPDATE, and DELETE statements. The only exception is when you specify a JSONP callback, in which case you can use an HTTP GET request and specify a callback query parameter on the GET URI. PHP SDK: From a Web application that uses the PHP SDK2, by calling the query method of the YahooSession class.

The Two-Minute Tutorial


This tutorial shows you how to run YQL statements and examine the resulting data with the YQL Console3. 1. In your browser, run the YQL Console4. 2. Under "Example Queries", click get 10 flickr "cat" photos. The console calls the YQL Web Service with the following query: select * from flickr.photos.search where text="Cat" limit 10 This SELECT statement requests Flickr photos of cats. The "*" indicates that all fields of the flickr.photos.search table will be returned. In the filter of the WHERE clause, the string "Cat" is the value of the search query.

1 2 3

http://developer.yahoo.com/yql/console/ http://developer.yahoo.com/social/sdk/#php http://developer.yahoo.com/yql/console/ 4 http://developer.yahoo.com/yql/console/

Yahoo! Developer Network

September 8, 2011

Overview for YQL Users

3. Note the XML response in the FORMATTED VIEW tab. The information from the flickr.photos.search table is in the results element. To get the response in JSON format, select the JSON radio button and click TEST. 4. In Your YQL Statement, replace the "*" with the owner and title fields: select owner, title from flickr.photos.search where text="Cat" limit 10 Make sure that owner and title are lowercase. Unlike SQL, in YQL the field and table names are case sensitive. 5. To run the command in Your YQL Statement, click TEST. The returned photo fields should only have the owner and title attribute fields. 6. In the console, examine the URL below REST query: http://query.yahooapis.com/v1/public/yql?q=select%20owne r % 2 C % 2 0 t i t l e % 2 0 f r o m % 2 0 f l i c k r . p h o tos.search%20where%20text%3D%22Cat%22%20limit%2010&format=json&callback=cbfunc To call the YQL Web Service, an application would call an HTTP GET method on this URL. The q parameter in the URL matches the SELECT statement displayed under Your YQL Statement (except that characters such as spaces are URL encoded). The COPY URL button copies this URL to your clipboard, so that you can paste it into the source code of an application. 7. To view YQL's pre-defined tables, expand the Data Tables list on the right side of the console. You can run an example query on each of these tables by clicking the table name. 8. Advanced: To view the description of a table, under Data Tables, expand the flickr menu to see all of the Flickr tables. Move your mouse cursor over the table name flickr.photos.search, then click desc. On the TREE VIEW tab, take a look at these nodes: "query->results->table". The nodes under the "table" node contain information such as meta-data and search fields (input keys). For more information on input keys, see Remote Filters [7].

YQL Web Service URLs


The YQL Web Service has two URLs. The following URL allows access to public data, which does not require authorization: http://query.yahooapis.com/v1/public/yql?[query_params] The next URL requires authorization by OAuth and allows access to both public and private data: http://query.yahooapis.com/v1/yql?[query_params] The following URLs are for accessing data from Open Data Tables configured to use YQL streaming5. The first URL is used to get public data, and the second requires authorization by OAuth and allows access to both public and private data. http://query.yahooapis.com/v1/public/streaming/yql

yql-odt-streaming.html

Yahoo! Developer Network

September 8, 2011

Overview for YQL Users

http://query.yahooapis.com/v1/streaming/yql

Note
The public URL has a lower rate limit than the OAuth-protected URL. Therefore, if you plan to use YQL heavily, you should access the OAuth-protected URL.

YQL Query Parameters


The following table lists the query parameters for the URLs of the YQL Web Service. Query Parameter Required? Default q format callback Yes No No xml Description The format of the results of the call to the YQL Web Service. Allowed values: xml or json.

(none) The YQL statement to execute, such as SELECT.

(none) The name of the JavaScript callback function for JSONP format. If callback is set and if format=json, then the response format is JSON. For more information on using XML instead of JSON, see JSONP-X [34]. true Diagnostic information is returned with the response unless this parameter is set to false.

diagnostics debug

No No

(none) Enables network-level logging of each network call within a YQL statement or API query. For more information, see, Logging Network Calls in Open Data Tables [12].

env jsonCompat

No No

(none) Allows you to use multiple Open Data Tables through a YQL environment file [42]. (none) Enables lossless JSON processing. The only allowed value is new. See JSON-to-JSON Transformation [38].

YQL Query Aliases


To make YQL Web Service queries easier to remember and share, you can shorten them with query aliases. Take for example a normal YQL REST query: http://query.yahooapis.com/v1/yql?q=select+*+from+weather.forecast+where+location%3D%40zip The corresponding query alias can look like this: http://query.yahooapis.com/v1/public/yql/jonathan/weather?zip=94025

Creating Queries Aliases


The easiest way to create query aliases is through the YQL console6, though you can also use some of the more advanced functions through the yql.queries7 and yql.queries.query8 tables.
6 7

http://developer.yahoo.com/yql/console/ http://developer.yahoo.com/yql/console/#h=desc%20yql.queries 8 http://developer.yahoo.com/yql/console/#h=desc%20yql.queries.query

Yahoo! Developer Network

September 8, 2011

Overview for YQL Users

To create a short query alias based on a statement, follow these steps: 1. Go the YQL console9. 2. Sign in with your Yahoo! ID if you haven't already. Click on the Sign In link, which is located along the top of the page. Signing in is required for saving and accessing query aliases. 3. Using the YQL console, create a YQL statement or Web Service query that you want to save as a query alias. 4. Click on the Create Query Alias link, which is located in the upper-right corner of the YQL statement box. 5. When first creating a query alias, type in a prefix that will be associated with all your query aliases. For example, if you choose "prefix," each query alias you create will start like this: http://query.yahooapis.com/v1/public/yql/prefix/

Tip
If you decide later that you want to start over with a new prefix for your query aliases, you can run the following YQL statement to delete your existing prefix: delete from yql.queries Keep in mind that deleting your prefix also deletes all its associated query aliases. 6. Click Next and then type the actual name of the query alias. This alias name is appended to the end of your query after the prefix. If you choose "alias", the resulting query will look like this: http://query.yahooapis.com/v1/public/yql/prefix/alias/ 7. After reviewing the public URL and the Authenticated URL, click Close to return to the YQL console. All of your saved query aliases appear along the right side the YQL console under the Query Aliases heading. There, you can click on a alias name to see it in the REST query box. You can also click on the red X next to a particular alias name to delete it.

Variable Substitution with Query Aliases


Query aliases support variable substitution [14] using the @ symbol. For example, the following YQL statement searches for music artists who contain the name Michael Jackson: select * from music.artist.search where keyword="Michael Jackson" Using variable substitution, you can replace Michael Jackson with the variable artist. Here is an example: select * from music.artist.search where keyword=@artist A corresponding query alias with the required parameter can look like this: http://query.yahooapis.com/v1/public/yql/prefix/musicartist?artist="Michael Jackson"
9

http://developer.yahoo.com/yql/console/

Yahoo! Developer Network

September 8, 2011

Overview for YQL Users

Summary of YQL Statements


The following table lists all YQL statements: Statement SELECT Example Description

SELECT * FROM social.pro- Retrieves data from the specified table. See the SELECT Statement chapter [6] for more infile WHERE guid=me formation. INSERT INTO table (key1, Inserts data into the specified table. See the INkey2, key3) VALUES SERT, UPDATE, DELETE statements [20] ('value1', 'value2', chapter for more information. 'value3') UPDATE (table) SET Updates data in the specified table. See the INfield1=value WHERE filter SERT, UPDATE, DELETE statements [20] chapter for more information. DELETE FROM (table) WHERE Deletes data in the specified table. See the INSERT, UPDATE, DELETE statements [20] filter chapter for more information. W SHOW TABLES DESC social.connections Gets a list of the tables available in YQL. Gets a description of the table.

INSERT

UPDATE

DELETE

S H O TABLES DESC USE

USE "http://myserv- Maps a table name to the URL of an Open Data er.com/mytables.xml" AS Table [41]. mytable; SELECT * WHERE... FROM mytable Allows you to set up key values [44] for use within Open Data Tables mytable

SET

SET (name)=(value); SELECT * WHERE... FROM

Authorization
A YQL table contains either public or private data. An example of public data is search information, such as the local.search table. An application can access a public table through the /v1/public/yql endpoint, which does not require authorization. (For the full endpoint, see YQL Web Service URLs [2].) A user's personal information, such as the social.contacts table, is private. Access to private data requires the user's approval. To access a private table, an application must use OAuth and the /v1/yql endpoint. YQL supports two-legged10 and three-legged11 OAuth. For YQL code examples with OAuth, see the YQL Code Examples [20]. For details, see the Yahoo! OAuth Quick Start Guide12 and the OAuth site13.

10 11 12

http://developer.yahoo.com/yos/glossary/gloss-entries.html#two-legged-authorization http://developer.yahoo.com/yos/glossary/gloss-entries.html#three-legged-authorization http://developer.yahoo.com/oauth/guide/index.html 13 http://oauth.net/

Yahoo! Developer Network

September 8, 2011

Chapter 2. SELECT Statement


In this Chapter Introduction to SELECT [6] Syntax of SELECT [6] Specifying the Elements Returned (Projection) [7] Filtering Query Results (WHERE) [7] Joining Tables With Sub-selects [10] Paging and Table Limits [10] Sort and Other Functions [12] Remote and Local Processing [12] GUIDs, Social, and Me [14] Variable Substitution in the GET Query String [14] Extracting HTML Content With XPath [15]

Introduction to SELECT
The SELECT statement of YQL retrieves data from YQL tables. The YQL Web Service fetches data from a back-end datasource (often a Web service), transforms the data, and returns the data in either XML or JSON format. Table rows are represented as repeating XML elements or JSON objects. Columns are XML sub-elements or attributes, or JSON name-value pairs. To try out some SELECT examples and to view the results, run the YQL Console1 and click the items under "Example Queries" or "Data Tables".

Syntax of SELECT
The YQL SELECT statement has the following syntax: SELECT what FROM table WHERE filter [| function] The what clause contains the fields (columns) to retrieve. The fields correspond to the XML elements or JSON objects in the data returned by the SELECT. An asterisk (the "*" character) in the what clause means all fields. The table is either the YQL pre-defined or Open Data Table that represents a datasource. (Unlike in SQL, in YQL only one table can be specified.) The filter is a comparison expression that limits the rows returned. The results of the SELECT can be piped to an optional function, such as sort. In YQL, statement keywords such as SELECT and WHERE are case-insensitive. Table and field names are case sensitive. In string comparisons, the values are case sensitive. String literals must be enclosed in quotes; either double or single quotes are allowed.

http://developer.yahoo.com/yql/console/

Yahoo! Developer Network

September 8, 2011

SELECT Statement

Specifying the Elements Returned (Projection)


To get a vertical slice (projection) of a table, specify the fields in the clause following the SELECT keyword. In YQL, these fields are analogous to the columns of a SQL table. Multiple fields are delimited by commas, for example: select lastUpdated, itemurl from social.updates where guid=me To get all fields, specify an asterisk: select * from social.updates where guid=me If the fields in the result set contain sub-fields, you can indicate the sub-fields by using periods (dots) as delimiters. (Sometimes this format is called "dot-style syntax.") For example, for the social.profile table, to get only the imageUrl sub-field of the image field, enter the following: select image.imageUrl from social.profile where guid=me The following lines show part of the XML response for this SELECT. Note that only the imageUrl subfield is returned.

. . . <results> <profile xmlns="http://social.yahooapis.com/v1/schema.rng"> <image> <imageUrl>http://l.yimg.com/us.yimg.com/i/identity/nopic_192.gif</imageUrl> </image> </profile> </results>

If you specify one or more non-existent fields in the what clause, the HTTP response code is 200 OK. If none of the fields in the what clause exist, the result set is empty. (That is, zero rows are returned.) Note that field names are case sensitive.

Filtering Query Results (WHERE)


The filter in the WHERE clause determines which rows are returned by the SELECT statement. The filter in the following statement, for example, returns rows only if the text field matches the string Barcelona: select * from flickr.photos.search where text='Barcelona' YQL has two types of filters: remote [7] and local [8]. These terms refer to where the filtering takes place relative to the YQL Web Service.

Remote Filters
With a remote filter, the filtering takes place in the back-end datasource (usually a Web service) called by the YQL Web Service. A remote filter has the following syntax:

Yahoo! Developer Network

September 8, 2011

SELECT Statement

input_key=literal The input key is a parameter that YQL passes to the back-end datasource. The literal is a value, either a string, integer, or float. Only the equality (=) operator is allowed in a remote filter. (A local filter [8], in contrast, can contain other types of comparison operators.) For example, in the following statement, the input key is photo_id: select * from flickr.photos.info where photo_id='2186714153' For this SELECT statement, the YQL Web Service calls the Flickr Web Service, passing photo_id as follows: http://api.flickr.com/services/rest/?method=flickr.photos.getInfo&photo_id='2186714153' Most YQL tables require the SELECT statement to specify a remote filter, which requires an input key. Often, the input key is not one of the fields included in the results returned by a SELECT. To see which input keys are allowed or required, enter the DESC statement for the YQL table and note the key XML element of the results. For example, the results of DESC flickr.photos.info show that the input key photo_id is required:

<results> . . . <select> <key name="secret" type="xs:string"/> <key name="photo_id" required="true" type="xs:string"/> </select> . . . <results>

Multiple remote filters can be combined with the boolean AND or OR operators, for example: select * from flickr.photos.info photo_id='3502889956' where photo_id='2186714153' or

The SELECT statements for some tables require multiple remote filters, for example: select * from local.search where zip='94085' and query='pizza'

Local Filters
The YQL Web Service performs local filtering on the data it retrieves from the back-end datasource. Before examining the syntax of local filters, let's look at a few examples. In the following example, YQL gets data from the flickr.photos.interestingness table, then applies the local filter title='moon'. select * from flickr.photos.interestingness where title='moon' In the next statement, the local filter checks that the value of the title field starts with the string Chinese or CHINESE.

Yahoo! Developer Network

September 8, 2011

SELECT Statement

select * from flickr.photos.interestingness where title like 'Chinese%' The filter in the following statement contains a regular expression that checks for the substring blue: select * from '.*blue.*' flickr.photos.interestingness where title matches

The following statement returns recent photos with the IDs specified in the parentheses: select * from flickr.photos.recent '3630791510', '3630791496') A local filter has the following syntax: field comparison_operator literal The field (column) specifies the name of the XML element or JSON object in the results. To specify a sub-field, separate the containing fields with periods. For an example sub-field, see Rating.AverageRating in the SELECT statement in Combining Boolean Operations [9]. The literal is either a quoted string, an integer, or a float. The following table lists the allowed comparison operators. Operator = != > < >= <= [NOT] IN IS [NOT] NULL [NOT] LIKE Equal. Not equal. Greater than. Less than. Greater than or equal to. Less than or equal to. Tests whether a value is contained in a set of values. This operator can be followed by either a sub-select or by a comma-delimited set of values within parentheses. Tests for the existence of the field in the results. An IS NULL expression is true if the field is not in the results. Tests for a string pattern match. The comparison is case-insensitive. The "%" character in the literal indicates zero or more characters. For example, Sys% matches any string starting with Sys. Description where id in ('3630791520',

[NOT] MATCHES Tests for a string pattern match, allowing regular expressions. The comparison is case sensitive.

Combining Filter Expressions (AND, OR)


Local and remote filter expressions can be combined with the boolean AND and OR operators. The AND operator has precedence over the OR operator. To change precedence, enclose expressions in parentheses. In the following example, the first two filters are remote expressions because query and location are input keys. The third filter, containing the field Rating.AverageRating, is a local filter. select * from local.search where query="sushi" and location="san francisco, ca" and Rating.AverageRating="4.5"

Yahoo! Developer Network

September 8, 2011

SELECT Statement

Joining Tables With Sub-selects


With sub-selects, you can join data across different YQL tables. (In SQL, a sub-select is usually called a "subquery.") Because YQL tables are often backed by Web services, sub-selects enable you to join data from different Web services. In a join, the sub-select provides input for the IN operator [9] of the outer select. The values in the outer select can be either input keys (remote filters [7]) or fields in the response (local filters [8]). By using a sub-select, the following statement returns the profiles of all of the connections (friends) of the user currently logged in to Yahoo!. This statement joins the social.profile and social.connection tables on the values of the GUIDs [14]. The inner SELECT, which follows the word IN, returns the GUIDs for the user's connections. For each of these GUIDs, the outer SELECT returns the profile information. select * from social.profile where guid in (select guid from social.connections where owner_guid=me) Tables can be joined on multiple keys. In the following example, the local.search and geo.places tables are joined on two keys. The inner select returns two data fields (centroid.latitude and centroid.latitude) which are compared with the two input keys (latitude and longitude) of the outer select. select * from local.search where (latitude,longitude) in (select centroid.latitude, centroid.longitude from geo.places where text="north beach, san francisco") and radius=1 and query="pizza" and location="" The next example shows an inner select that returns data from an RSS feed: select * from search.web where query in (select title from rss where url="http://rss.news.yahoo.com/rss/topstories" | truncate(count=1)) One sub-select is allowed in each select. In other words, each select statement can only have one IN keyword, but the inner select may also have an IN keyword. The following statement is legal: select * from search.siteexplorer.pages where query in (select url from search.web where query in (select Artist.name from music.release.popular limit 1) limit 1) However, the next statement is illegal because it has two IN keywords in a select: ILLEGAL: select * from flickr.photos.search where lat in (select centroid.latitude from geo.places where text="sfo") and lon in (select centroid.longitude from geo.places where text="sfo")

Paging and Table Limits


Many YQL queries access datasources that contain thousands, or even millions, of items. When querying large datasources, applications need to page through the results to improve performance and usability. YQL enables applications to implement paging or limit table size at two levels: remote [11] and local [11]. To find out how many items (rows) a query (SELECT) returns, in an XML response, check the value of the yahoo:count attribute of the query element. In a JSON response, check the value of the count object.

Yahoo! Developer Network

10

September 8, 2011

SELECT Statement

The maximum number of items returned by a SELECT is 5000. The maximum processing time for a YQL statement is 30 seconds. For most tables, the default number of items returned is 10. (That is, the default is 10 if you do not specify a limit in the SELECT statement.)

Remote Limits
A remote limit controls the number of items (rows) that YQL retrieves from the back-end datasource. To specify a remote limit, enter the offset (start position) and number of items in parentheses after the table name. For example, in the following statement, the offset is 0 and the number of items is 10. When this statement runs, YQL calls Yahoo! Search BOSS (the back-end source for the search.web table) and gets the first 10 items that match the query="pizza" filter: select title from search.web(0,10) where query="pizza" The following statement gets items 10 through 30. In other words, starting at postition 10, it gets 20 items: select title from search.web(10,30) where query="pizza" The default offset is 0. For example, the following statement gets the first 20 items: select title from search.web(20) where query="pizza" The default number of items for a remote limit varies with the table. For most tables, the default number of items is 10. The maximum number of items also varies with table. To get the maximum number of items, enter 0 in parentheses after the table name. The following statement returns 1000 items from the search.web table: select title from search.web(0) where query="pizza"

Local Limits
A local limit controls the number of rows YQL returns to the calling application. YQL applies a local limit to the data set that it has retrieved from the back-end datasource. To specify a local limit, include the LIMIT and OFFSET keywords (each followed by an integer) after the WHERE clause. LIMIT specifies the number of rows and OFFSET indicates the starting position. The OFFSET keyword is optional. The default offset is 0, which is the first row. The following statement has a remote limit of 100 and a local limit of 15. When this statement runs, YQL gets up to 100 items from the back-end datasource. On these items, YQL applies the local limit and offset. This statement returns 15 rows to the calling application, starting with the first row (offset 0). select title from search.web(100) where query="pizza" limit 15 offset 0 YQL retrieves items from the back-end datasource one page at a time until either the local or remote limit has been reached. The page size varies with the table. The following statement has an unbounded remote limit (0) so YQL retrieves items from the backend datasource until the the local limit of 65 is reached: select title from search.web(0) where query="pizza" limit 65

Yahoo! Developer Network

11

September 8, 2011

SELECT Statement

Typically, a SELECT statement includes limits and filters, as shown in Example With Remote and Local Steps [13].

Sort and Other Functions


YQL includes built-in functions such as sort, which are appended to the SELECT statement with the pipe symbol ("|"). These functions are applied to the result set after the SELECT statement performs all other operations, such as applying filters and limits. In the following SELECT statement, the sub-select returns a list of GUIDs, and the outer select returns a set of profiles, one for each GUID. This set of profiles is piped to the sort function, which orders the results according to the value of the nickname field. select * from social.profile where guid in (select guid from social.connections where owner_guid=me) | sort(field="nickname") Multiple functions can be chained together with the pipe symbol ("|"). The following statement queries the local.search table for restaurants serving pizza. The results are piped to the sort function, then to the reverse function. The final result contains 20 rows, sorted by rating from high to low. select Title, Rating.AverageRating from local.search(20) where query="pizza" and city="New York" and state="NY" | sort(field="Rating.AverageRating") | reverse() The following table lists the YQL functions that can be appended to a SELECT statement. Function arguments are specified as name-value pairs. Function sort Argument Example Description

field [des- sort(field="nick- Sorts the result set according to the specified cending] name", descend- field (column) in the result set. The default value of the optional descending arguing="true") ment is false. count (none) field tail(count=4) truncate(count=4) reverse() Gets the last count items (rows). Gets the first count items (rows). Reverses the order of the items (rows).

tail reverse unique

truncate count

unique(field="Rat- Removes items (rows) with duplicate values ing.AverageRating") in the specified field (column). The first item with the value remains in the results. s a n i t ize(field='foo') Sanitizes the output for HTML-safe rendering. To sanitize all returned fields, omit the field parameter.

sanitize [field]

Remote and Local Processing


When YQL runs a SELECT statement, it accesses a back-end datasource, typically by calling a Web service. Remote filters [7] and limits [11] are implemented by the back-end Web service. Local processing (including local filters [8] and limits [11]) is performed by the YQL Web Service on the data it fetches from the back-end Web service. As shown by the following example, whether an operation is remote or local affects the data returned to the application that calls the SELECT statement.

Yahoo! Developer Network

12

September 8, 2011

SELECT Statement

Example With Remote and Local Steps


The following SELECT statement gets data about pizza restaurants from the search.web table:

select from where limit

title, abstract, url search.web(500) (query='pizza') and ((title like 'Round%') or (abstract matches 5 | sort(field='title')

'.*about.*'))

The steps that follow show the order in which YQL processes the remote and local parts of the SELECT statement: 1. YQL calls Yahoo! Search BOSS, the Web service behind the search.web table, at the following URL: h t t p : / / b o s s . y a h o o is.com/ysearch/web/v1/pizza?format=xml&start=0&count=50 a p -

By calling this URL, YQL gets the first 50 items that match the SELECT statement's remote filter: query='pizza'. The query element is an input key for the search.web table. Although the remote filter in the SELECT is set to 500, to improve efficiency, YQL only fetches 50 items each time it calls BOSS. 2. On the 50 items it retrieved from BOSS, YQL applies the local filter: ((title like 'Round%') or (abstract matches '.*about.*')). This filter selects an item if the title field (column) starts with Round or the abstract field contains about. In this example, from the set of 50 items, YQL finds 3 items that match the local filter. 3. YQL checks that the items retrieved from BOSS contain the title , abstract, and url fields, which are specified after the SELECT keyword. 4. YQL calls BOSS again, incrementing the start parameter to 50, to get the next 50 rows: h t t p : / / b o s s . y a h o o is.com/ysearch/web/v1/pizza?format=xml&start=50&count=50 On this second set of 50 items, YQL applies the local filter, but finds no matches. 5. YQL calls BOSS a third time to get the next 50 items: h t t p : / / b o s s . y a h o o a is.com/ysearch/web/v1/pizza?format=xml&start=100&count=50 p a p -

6. On the third set of items from BOSS, YQL applies the local filter and verifies that the title, abstract, and url fields exist. This time, YQL finds 2 more matches, which brings the total number of matches to 5. Because the local limit of 5 has been reached, YQL does not call BOSS again. 7. YQL pipes the 5 items to the sort function, ordering the data by the title field. 8. YQL returns 5 rows (containing just the title, abstract, and url fields) to the calling Web application.

Yahoo! Developer Network

13

September 8, 2011

SELECT Statement

Summary of Remote and Local Controls


The following table identifies whether an element in the SELECT statement is processed locally or remotely by YQL. Syntax Element in SELECT Columns or asterisk after the SELECT keyword. Local or Remote Local Section With Details Specifying the Elements Returned (Projection) [7] Remote Limits [11] Remote Filters [7]

Remote limit and offset, indicated by integers in paren- Remote theses after the table name. Remote filter expression in the WHERE clause. The only Remote allowed operator is an equal sign. The value compared is an input key for the back-end datasource. Local filter expression in the WHERE clause. Various Local operators are allowed, including LIKE and MATCH. The value compared is a field (column) in the data returned by the query. LIMIT and OFFSET keywords after the WHERE clause. Local Sort and other functions after the pipe (|) symbol. Local

Local Filters [8]

Local Limits [11] Sort and Other Functions [12]

GUIDs, Social, and Me


YQL includes a set of pre-defined tables that call the Yahoo! Social APIs. The social.profile table, for example, contains information about a Yahoo! user, and the social.connections table is a list of the user's friends. The Global User Identifier (GUID) is a string that uniquely identifies a Yahoo! user. In YQL, the me keyword is the GUID value of the user currently logged in to Yahoo!. For example, if you are logged in to Yahoo!, and you run the following statement, YQL returns your profile information: select * from social.profile where guid=me Because me is a keyword, it is not enclosed in quotes. To specify a GUID value, enclose the string in quotes, for example: select * from social.updates where guid='7WQ7JILMQKTSTTURDDAF3NT35A'

Variable Substitution in the GET Query String


If the URL contains @var literals, YQL replaces the literals with the values of query parameters with the same names. For example, suppose that the URL for the call to the YQL Web Service has the animal query parameter: http://query.yahooapis.com/v1/yql?animal=dog&q=select * from sometable where animal=@animal For this URL, YQL will run the following SELECT statement: select * from sometable where animal="dog"

Yahoo! Developer Network

14

September 8, 2011

SELECT Statement

Extracting HTML Content With XPath


A key feature of YQL is the ability to access data from structured data feeds such as RSS and ATOM. However, if no such feed is available, you can specify the source as HTML and use XPath to extract the relevant portions of the HTML page. For example, to get information from Yahoo! Finance about Yahoo! Inc. stock (YHOO), you might start with the following YQL statement: select * from html where url="http://finance.yahoo.com/q?s=yhoo" Run this example in the YQL Console2 Because the preceding statement returns all of the page's HTML, it would not be useful in an application. By adding an XPath expression to the statement, you can retrieve specific portions of the HTML page. The XPath expression in the following statement traverses through the nodes in the HTML page to isolate the latest headlines. In this case, the XPath expression looks first for a div tag with the ID yfi_headlines. Next, the expression gets the second div tag and looks for an anchor tag (a) within a list item (li) of an unordered list (ul). Here's the YQL statement with the XPath expression: select * from html where url="http://finance.yahoo.com/q?s=yhoo" and xpath='//div[@id="yfi_headlines"]/div[2]/ul/li/a' Run this example in the YQL console3 The following statement also gets information about Yahoo! Inc. stock, but traverses the nodes to get key statistics: select * from html where url="http://finance.yahoo.com/q?s=yhoo" and xpath='//div[@id="yfi_key_stats"]/div[2]/table' Instead of the the wildcard asterisk (*), you can specify a particular element for the XPath to process. For example, the following statement extracts only the HTML links (href tags) within the headlines on Yahoo! Finance: select href from html where url="http://finance.yahoo.com/q?s=yhoo" and xpath='//div[@id="yfi_headlines"]/div[2]/ul/li/a' Run this example in the YQL console4 To get just the content from an HTML page, you can specify content keyword after the word select. A statement with the content keyword processes the HTML in the following order: 1. It looks for any element named "content" within the elements found by the XPath expression. 2. If an element named "content" is not found, the statement looks for an attribute named "content".

http://developer.yahoo.com/yql/console/?q=select%20*%20from%20html%20where%20url%3D%22http%3A%2F%2Ffinance.yahoo.com%2Fq%3Fs%3Dyhoo%22 3 http://developer.yahoo.com/yql/console/?q=select%20*%20from%20html%20where%20url%3D%22http%3A%2F%2Ffinance.yahoo.com%2Fq%3Fs%3Dyhoo%22%20and%0A%20%20%20%20%20%20xpath%3D%27%2F%2Fdiv[%40id%3D%22yfi_headlines%22]%2Fdiv[2]%2Ful%2Fli%2Fa%27%0A 4 http://developer.yahoo.com/yql/console/?q=select%20href%20from%20html%20where%20url%3D%22http%3A%2F%2Ffinance.yahoo.com%2Fq%3Fs%3Dyhoo%22%20and%20xpath%3D%27%2F%2Fdiv[%40id%3D%22yfi_headlines%22]%2Fdiv[2]%2Ful%2Fli%2Fa%27

Yahoo! Developer Network

15

September 8, 2011

SELECT Statement

3. If neither an element nor attribute named "content" is found, the statement returns the element's textContent. The following statement, for example, returns the textContent of each anchor (a) tag retrieved by the XPath expression: select content from html where url="http://finance.yahoo.com/q?s=yhoo" and xpath='//div[@id="yfi_headlines"]/div[2]/ul/li/a' Run this example in the YQL console5

http://developer.yahoo.com/yql/console/?q=select%20content%20from%20html%20where%20url%3D%22http%3A%2F%2Ffinance.yahoo.com%2Fq%3Fs%3Dyhoo%22%20and%20xpath%3D%27%2F%2Fdiv[%40id%3D%22yfi_headlines%22]%2Fdiv[2]%2Ful%2Fli%2Fa%27

Yahoo! Developer Network

16

September 8, 2011

Chapter 3. Using the Query Builder


In this Chapter YQL Query Builder Overview [17] Query Builder Usage [18]

YQL Query Builder Overview


To aid in building YQL statements (queries), the YQL console includes a simple query builder tool. This tool allows you to customize and explore YQL statements.

Figure 3.1. YQL Query Builder

Note
This simple query builder is intended to explore and test simple statements. It does not currently support more advanced statements that include sub-queries, multiple Open Data Tables, or projection. Features of YQL query builder include: Tabs for each table within a statement: Each Open Data Table used within a given statement is shown as as a separate tab. Limit and tail functions: Easily add functions to get the first or last number of results.

Yahoo! Developer Network

17

September 8, 2011

Using the Query Builder

Table metadata: Click "show table info" to see metadata within each Open Data Table.

Query Builder Usage


This section describes how to use the various aspects of the query builder.

Loading a Query
1. To run the query builder, first test a YQL statement in the console. Below the statement box and above the results, a tab appears for each Open Data Table that is required to complete the statement.

2. Click on a tab associated with an Open Data Table to load the query builder for that table.

Yahoo! Developer Network

18

September 8, 2011

Using the Query Builder

Entering Values
After you load a query, you are presented with a form that contains fields for each required binding. 1. Type values for each required field. 2. Below the required bindings, you can also click Add Key to add optional bindings to the statement. Click on the X icon next to a binding to remove it.

3. Click TEST within the query builder to run the updated YQL statement.

Yahoo! Developer Network

19

September 8, 2011

Chapter 4. INSERT, UPDATE, and DELETE (I/U/D) Statements


In this Chapter: Introduction to INSERT, UPDATE, and DELETE (I/U/D) Statements [20] Bindings Required for I/U/D [20] JavaScript Methods Available to I/U/D [22] Syntax of I/U/D [22] Limitations of I/U/D [23] Open Data Table Examples of I/U/D [24]

Introduction to INSERT, UPDATE, and DELETE (I/U/D) Statements


While YQL SELECT statements allow you to read structured data from almost any source on the Web, their expressed purpose is only to read data. To perform data manipulation, YQL provides three other SQL-like keywords for writing, updating, and deleting data mapped using a YQL Open Data Table, namely INSERT, UPDATE, and DELETE (I/U/D). The INSERT statement inserts or adds new data to YQL tables, while the UPDATE statement updates or modifies existing data. DELETE, as the name implies, removes data. I/U/D statements require the proper binding inputs [6], such as key, value, or map. The actual addition, modification, or deletion of data is performed within the Open Data Table [41].

Caution
Most sources that provide write capability need authentication. Examples of authentication include username/password combinations or secret API tokens. If your table requires input that is deemed "private", such as any passwords, authentication keys, or other "secrets", you MUST ensure the https attribute within the tables element is set to true. For more information on about securing private data in Open Data Tables, refer to Ensuring the Security of Private Information [20].

Bindings Required for I/U/D


I/U/D statements rely entirely on appropriate bindings within an Open Data Table to be usable. Specifically, you must use an insert, update, or delete bindings element [3]. These elements help to determine what happens with the information you pass in through a YQL statement.

Yahoo! Developer Network

20

September 8, 2011

INSERT, UPDATE, and DELETE (I/U/D) Statements Consider the following INSERT statement for shortening URLs using bit.ly: INSERT INTO bitly.shorten (login, apiKey, longUrl) VALUES ('USERNAME', 'API_KEY', 'http://yahoo.com') The corresponding Open Data Table for this statement follows: <?xml version="1.0" encoding="UTF-8"?> <table xmlns="http://query.yahooapis.com/v1/schema/table.xsd" https="true"> <meta> <author>Nagesh Susarla</author> <documentationURL>http://code.google.com/p/bitly-api/wiki/ApiDocumentation</documentationURL> </meta> <bindings> <insert itemPath="" produces="XML"> <urls> <url>http://api.bit.ly/shorten?version=2.0.1&amp;format=xml&amp;{-join|&amp;|longUrl}</url> </urls> <inputs> <key id="login" type="xs:string" paramType="query" required="true"/> <key id="apiKey" type="xs:string" paramType="query" required="true"/> <value id="longUrl" type="xs:string" paramType="path" required="true"/> </inputs> </insert> </bindings> </table> Run this example on the YQL console1

Note
To run the example, you must replace USERNAME and API_KEY in the YQL statement with your actual bit.ly username and API key (available through your bit.ly2 account page). The above Open Data Table shows one of the most basic ways to use an INSERT statement because it does not require JavaScript to massage the data. Consequently, it requires no execute [19] element. It simply uses the login and apiKey as keys to authenticate the user, with the longUrl as the new value passed to the bit.ly API. For more information on how the key, value, and map values are used, refer to key/value/map elements in Using Open Data Tables [6].

https://developer.yahoo.com/yql/console/?q=USE%20%22http%3A%2F%2Fwww.yqlblog.net%2Fsamples%2Fbitly.shorten.xml%22%3B%20INSERT%20INTO%20bitly.shorten%20%28login%2C%20apiKey%2C%20longUrl%29%20VALUES%20%28%27YOUR_LOGIN%27%2C%20%27YOUR_API_KEY%27%2C%20%27http%3A%2F%2Fyahoo.com%27%29 2 http://bit.ly/

Yahoo! Developer Network

21

September 8, 2011

INSERT, UPDATE, and DELETE (I/U/D) Statements

Note
In the above example, the url element contains a URI template to aid in the construction of the URI: {-join|&amp;|longUrl} Here the join operator creates a key-value pair using the variable longURL along with its value. This pair is preceded by the ampersand symbol (&).

JavaScript Methods Available to I/U/D


For Web services that require specific authentication methods or specific types of HTTP requests, YQL provides several JavaScript methods for use within the execute [19] element: Methods that allow HTTP PUT, POST, and DELETE requests, in addition to GET. The ability to specify the content type on data being sent, using contentType. The ability to automatically convert the data being returned using accept. For more information on the JavaScript methods available for use within I/U/D statements, refer to the JavaScript Objects, Methods, and Variables Reference [20].

Syntax of I/U/D
This section discusses the syntax for I/U/D statements.

Syntax of INSERT
The YQL INSERT statement has the following syntax: INSERT INTO (table) (list of comma separated field names) VALUES (list of comma separated values) The INSERT INTO keywords marks the start of an INSERT statement. The table is either the YQL pre-defined or Open Data Table that represents a data source. Following the table name is a list of field names indicating the table columns where YQL inserts a new row of data. The VALUES clause indicates the data inserted into those columns. String values are enclosed in quotes. In YQL, statement keywords such as SELECT and WHERE are case-insensitive. Table and field names are case sensitive. In string comparisons, the values are case sensitive. String literals must be enclosed in quotes; either double or single quotes are allowed.

Syntax of UPDATE
The YQL UPDATE statement has the following syntax: UPDATE (table) SET field=value WHERE filter

Yahoo! Developer Network

22

September 8, 2011

INSERT, UPDATE, and DELETE (I/U/D) Statements The UPDATE keyword marks the start of an UPDATE statement. This is followed by the table name. The table is either the YQL pre-defined or Open Data Table that represents a data source. The SET clause is the part of the statement in which we pass new data to the update binding in the Open Data Table. The WHERE clause indicates which data should be updated. Only remote filters can be present in the WHERE clause of an UPDATE statement. The following example shows how the UPDATE statement syntax can look for updates to your status on Yahoo! Profiles: UPDATE social.profile.status SET status="Using YQL UPDATE" WHERE guid=me Try this example in the YQL console3 In the above example, status and guid are all bindings within the inputs element, which is nested within an update element. The status is a value element, since this is data that is updating a value using the Open Data Table. The guid binding is simply a key element, as it is a required "key" that determines ownership of this status.

Syntax of DELETE
The YQL DELETE statement has the following syntax: DELETE FROM [table] WHERE filter The DELETE keyword marks the start of a DELETE statement. The table is either the YQL pre-defined or Open Data Table that represents a data source. This is immediately followed by a remote filter that determines what table rows to remove. The following example deletes a particular Yahoo! update: DELETE FROM social.updates source="agg.bebo" WHERE guid="me" and suid="12345" and

In the example above, the remote filters are the SUIDof the update followed by the guid of the Yahoo! user and the source of the update.

Limitations of I/U/D
While I/U/D statements support most of the same functionality as the SELECT statement, there are a few caveats to keep in mind: Local filtering: I/U/D statements do not support local filtering [8]. Sub-selects: INSERT statements do not support sub-selects [10].

http://developer.yahoo.com/yql/console/?q=UPDATE%20social.profile.status%20SET%20status%3D%22Using%20YQL%20UPDATE%22%20WHERE%20guid%3Dme

Yahoo! Developer Network

23

September 8, 2011

INSERT, UPDATE, and DELETE (I/U/D) Statements Paging: I/U/D statements do not support paging [10]. However, you can use sub-selects within UPDATE and DELETE statements to narrow down the values you wish to insert or delete, as shown in the following example: DELETE FROM table WHERE guid IN (SELECT guid FROM social.connections WHERE owner_guid = me)

Open Data Table Examples of I/U/D


The following Open Data Tables provide examples of INSERT: Bit.ly Shorten URL [24] WordPress Post (Insert) [25]

Tip
To better understand the examples presented in this section, refer first to Using YQL Open Data Tables [41] and Executing JavaScript in Open Data Tables [19].

Bit.ly Shorten URL (INSERT)


The following Open Data Table allows you to shorten a long URL into a bit.ly link. This table showcases the following: using INSERT statements using value bindings Example Statement: INSERT INTO bitly.shorten (login, apiKey, longUrl) VALUES ('USERNAME', 'API_KEY', 'http://yahoo.com') Open Data Table Source: <?xml version="1.0" encoding="UTF-8"?> <table xmlns="http://query.yahooapis.com/v1/schema/table.xsd" https="true"> <meta> <author>Nagesh Susarla</author> <documentationURL>http://code.google.com/p/bitly-api/wiki/ApiDocumentation</documentationURL> </meta> <bindings> <insert itemPath="" produces="XML"> <urls> <url>http://api.bit.ly/shorten?version=2.0.1&amp;format=xml&amp;{-join|&amp;|longUrl}</url> </urls> <inputs>

Yahoo! Developer Network

24

September 8, 2011

INSERT, UPDATE, and DELETE (I/U/D) Statements <key id="login" type="xs:string" paramType="query" required="true"/> <key id="apiKey" type="xs:string" paramType="query" required="true"/> <key id="longUrl" type="xs:string" paramType="path" required="true"/> </inputs> </insert> </bindings> </table> Run this example on the YQL console4

Note
To run the example, you must replace USERNAME and API_KEY in the YQL statement with your actual bit.ly username and API key (available through your bit.ly5 account page).

WordPress Post (INSERT)


The following Open Data Table allows you to post to your WordPress blog. This table showcases the following: calling a YQL query within execute allowing INSERT in addition to SELECT statements performing HTTP POST with JavaScript methods creating an XML response with E4X6 Example Statement: USE "http://www.datatables.org/wordpress/wordpress.post.xml"; INSERT into wordpress.post (title, description, blogurl, username, password) values ("Test Title", "This is a test body", "http://your_wordpress_blog_site.com", "your_wordpress_username", "your_wordpress_password") Open Data Table Source: <?xml version="1.0" encoding="UTF-8"?> <table xmlns="http://query.yahooapis.com/v1/schema/table.xsd"> <meta> <sampleQuery>insert into {table} (title, description, blogurl, username, password) values ("Test Title", "This is a test body", "http://yqltest.wordpress.com", "yqltest", "password")</sampleQuery> </meta> <bindings>
4

https://developer.yahoo.com/yql/console/?q=USE%20%22http%3A%2F%2Fwww.yqlblog.net%2Fsamples%2Fbitly.shorten.xml%22%3B%20INSERT%20INTO%20bitly.shorten%20%28login%2C%20apiKey%2C%20longUrl%29%20VALUES%20%28%27YOUR_LOGIN%27%2C%20%27YOUR_API_KEY%27%2C%20%27http%3A%2F%2Fyahoo.com%27%29 5 http://bit.ly/ 6 http://en.wikipedia.org/wiki/ECMAScript_for_XML

Yahoo! Developer Network

25

September 8, 2011

INSERT, UPDATE, and DELETE (I/U/D) Statements <insert itemPath="" produces="XML"> <urls> <url></url> </urls> <inputs> <key id='username' type='xs:string' paramType='variable' required="true" /> <key id='password' type='xs:string' paramType='variable' required="true" /> <key id='blogurl' type='xs:string' paramType='variable' required="true" /> <value id='allow_comments' type='xs:string' paramType='variable' required="false" default="1" /> <value id='tags' type='xs:string' paramType='variable' required="false"/> <value id='keywords' type='xs:string' paramType='variable' required="false"/> <value id='convert_breaks' type='xs:string' paramType='variable' required="false" default="0" /> <value id='title' type='xs:string' paramType='variable' required="true" /> <value id='excerpt' type='xs:string' paramType='variable' required="false"/> <value id='description' type='xs:string' paramType='variable' required="true"/> <value id='moretext' type='xs:string' paramType='variable' required="false"/> <value id='allow_pings' type='xs:string' paramType='variable' required="false" default= "0" /> <value id='publish' type='xs:string' paramType='variable' required="false" default= "1" /> </inputs> <execute><![CDATA[ if (allow_comments == "true") { allow_comments = 1; } else if (allow_comments == "false") { allow_comments = 0; } if (convert_breaks == "true") { conver_tbreaks = 1; } else if (convert_breaks == "false") { convert_breaks = 0; } if (allow_pings == "true") { allow_pings = 1; } else if (allow_pings == "false") { allow_pings = 0; } postData = <methodCall><methodName>metaWeblog.newPost</methodName> <params> <param> <value> <string>1</string>

Yahoo! Developer Network

26

September 8, 2011

INSERT, UPDATE, and DELETE (I/U/D) Statements </value> </param> <param> <value> <string>{username}</string> </value> </param> <param> <value> <string>{password}</string> </value> </param> <param> <value> <struct> <member> <name>mt_allow_comments</name> <value> <int>{allow_comments}</int> </value> </member> <member> <name>mt_convert_breaks</name> <value> <string>{convert_breaks}</string> </value> </member> <member> <name>title</name> <value> <string>{title}</string> </value> </member> <member> <name>description</name> <value> <string>{description}</string> </value> </member> <member> <name>mt_allow_pings</name> <value> <string>{allow_pings}</string> </value> </member> </struct> </value> </param> <param> <value> <boolean>{publish}</boolean> </value> </param> </params>

Yahoo! Developer Network

27

September 8, 2011

INSERT, UPDATE, and DELETE (I/U/D) Statements </methodCall>; if (tags) { postData.params.param..struct.member += <member> <name>mt_tags</name> <value> <string>{tags}</string> </value> </member>; } else { postData.params.param..struct.member += <member> <name>mt_tags</name> <value> <string></string> </value> </member>; } if (moretext) { postData.params.param..struct.member += <member> <name>mt_text_more</name> <value> <string>{moretext}</string> </value> </member>; } else { postData.params.param..struct.member += <member> <name>mt_text_more</name> <value> <string></string> </value> </member>; } if (excerpt) { postData.params.param..struct.member += <member> <name>mt_excerpt</name> <value> <string>{excerpt}</string> </value> </member>; } else { postData.params.param..struct.member += <member> <name>mt_excerpt</name> <value> <string></string> </value> </member>; } if (keywords) { postData.params.param..struct.member +=

Yahoo! Developer Network

28

September 8, 2011

INSERT, UPDATE, and DELETE (I/U/D) Statements <member> <name>mt_keywords</name> <value> <string>{keywords}</string> </value> </member>; } else { postData.params.param..struct.member += <member> <name>mt_keywords</name> <value> <string></string> </value> </member>; } url = blogurl + "/xmlrpc.php"; myRequest = y.rest(url); myRequest.contentType("text/xml"); //myRequest.accept("text/xml"); myRequest.header("Connection", "close"); results = myRequest.post('<?xml version="1.0" encoding="UTF-8"?>' + postData.toString()).response; postId = results.params.param.value.string.toString(); //response.object = results; response.object = <postid>{postId}</postid>; ]]> </execute> </insert> <update itemPath="" produces="XML"> <urls> <url></url> </urls> <inputs> <key id='username' type='xs:string' paramType='variable' required="true" /> <key id='password' type='xs:string' paramType='variable' required="true" /> <key id='blogurl' type='xs:string' paramType='variable' required="true" /> <key id='postid' type='xs:string' paramType='variable' required="true" /> <value id='allow_comments' type='xs:string' paramType='variable' required="true" /> <value id='tags' type='xs:string' paramType='variable' required="true"/> <value id='keywords' type='xs:string' paramType='variable' required="true"/> <value id='convert_breaks' type='xs:string' paramType='variable' required="true" /> <value id='title' type='xs:string' paramType='variable' required="true" /> <value id='excerpt' type='xs:string' paramType='variable' required="true"/> <value id='description' type='xs:string' paramType='variable'

Yahoo! Developer Network

29

September 8, 2011

INSERT, UPDATE, and DELETE (I/U/D) Statements required="true"/> <value id='moretext' type='xs:string' paramType='variable' required="true"/> <value id='allow_pings' type='xs:string' paramType='variable' required="true" /> <value id='publish' type='xs:string' paramType='variable' required="true"/> </inputs> <execute> <![CDATA[ if (allow_comments == "true") { allow_comments = 1; } else if (allow_comments == "false") { allow_comments = 0; } if (convert_breaks == "true") { conver_tbreaks = 1; } else if (convert_breaks == "false") { convert_breaks = 0; } if (allow_pings == "true") { allow_pings = 1; } else if (allow_pings == "false") { allow_pings = 0; } postData = <methodCall> <methodName>metaWeblog.newPost</methodName> <params> <param> <value> <string>1</string> </value> </param> <param> <value> <string>{username}</string> </value> </param> <param> <value> <string>{password}</string> </value> </param> <param> <value> <struct> <member> <name>mt_allow_comments</name> <value> <int>{allow_comments}</int> </value> </member> <member> <name>mt_convert_breaks</name>

Yahoo! Developer Network

30

September 8, 2011

INSERT, UPDATE, and DELETE (I/U/D) Statements <value> <string>{convert_breaks}</string> </value> </member> <member> <name>title</name> <value> <string>{title}</string> </value> </member> <member> <name>description</name> <value> <string>{description}</string> </value> </member> <member> <name>mt_allow_pings</name> <value> <string>{allow_pings}</string> </value> </member> </struct> </value> </param> <param> <value> <boolean>{publish}</boolean> </value> </param> </params> </methodCall>; if (tags) { postData.params.param..struct.member += <member> <name>mt_tags</name> <value> <string>{tags}</string> </value> </member>; } else { postData.params.param..struct.member += <member> <name>mt_tags</name> <value> <string></string> </value></member>; } if (moretext) { postData.params.param..struct.member += <member> <name>mt_text_more</name> <value> <string>{moretext}</string>

Yahoo! Developer Network

31

September 8, 2011

INSERT, UPDATE, and DELETE (I/U/D) Statements </value> </member>; } else { postData.params.param..struct.member += <member> <name>mt_text_more</name> <value> <string></string> </value> </member>; } if (excerpt) { postData.params.param..struct.member += <member> <name>mt_excerpt</name> <value> <string>{excerpt}</string> </value> </member>; } else { postData.params.param..struct.member += <member> <name>mt_excerpt</name> <value> <string></string> </value> </member>; } if (keywords) { postData.params.param..struct.member += <member> <name>mt_keywords</name> <value> <string>{keywords}</string> </value> </member>; } else { postData.params.param..struct.member += <member> <name>mt_keywords</name> <value> <string></string> </value> </member>; } ]]> </execute> </update> <select itemPath="" produces="XML"> <urls> <url></url> </urls> <inputs> <key id='postid' type='xs:string' paramType='variable'

Yahoo! Developer Network

32

September 8, 2011

INSERT, UPDATE, and DELETE (I/U/D) Statements required="true" /> <key id='blogurl' type='xs:string' paramType='variable' required="true" /> <key id='username' type='xs:string' paramType='variable' required="true" /> <key id='password' type='xs:string' paramType='variable' required="true" /> </inputs> <execute><![CDATA[ postData = <methodCall> <methodName>metaWeblog.getPost</methodName> <params> <param> <value> <string>{postid}</string> </value> </param> <param> <value> <string>{username}</string> </value> </param> <param> <value> <string>{password}</string> </value> </param> </params> </methodCall>; url = blogurl + "/xmlrpc.php"; myRequest = y.rest(url); myRequest.contentType("text/xml"); //myRequest.accept("text/xml"); myRequest.header("Connection", "close"); results = myRequest.post('<?xml version="1.0" encoding="UTF-8"?>' + postData.toString()).response; response.object = results; ]]></execute> </select> </bindings> </table> Run this example7 in the YQL Console. Be sure to use your blog's URL, username, and password as values for the blogurl, username, and password keys.

https://developer.yahoo.com/yql/console/?q=select%20*%20from%20twitter.account.credentials;&env=store://datatables.org/alltableswithkeys#h=USE%20%22http%3A//www.datatables.org/wordpress/wordpress.post.xml%22%3B%20INSERT%20into%20wordpress.post%20%28title%2C%20description%2C%20blogurl%2C%20username%2C%20password%29%20values%20%28%22Test%20Title%22%2C%20%22This%20is%20a%20test%20body%22%2C%20%22http%3A//your_wordpress_blog_site.com%22%2C%20%22your_wordpress_username%22%2C%20%22your_wordpress_password%22%29

Yahoo! Developer Network

33

September 8, 2011

Chapter 5. Response Data


In this Chapter: Supported Response Formats [34] JSONP-X: JSON envelope with XML content [34] Structure of Response [36] Information about the YQL Call [36] XML-to-JSON Transformation [37] Errors and HTTP Response Codes [39]

Supported Response Formats


The YQL Web Service can return data in either XML, JSON, or JSONP format. The default format is XML. To specify JSON, include the format=json parameter in the URL of the YQL Web service, for example: http://query.yahooapis.com/v1/public/yql?q=select * from social.connections where owner_guid=me&format=json To specify JSONP, include both the format and callback query parameters. The callback parameter indicates the name of the JavaScript callback function. Here's an example: http://query.yahooapis.com/v1/public/yql?q=select * from social.connections where owner_guid=me&format=json&callback=cbfunc The format of the response data is not dependent on the format of the original datasource. For example, if a YQL table is backed by a datasource in XML, the YQL Web Service can return data in JSON. For more information, see XML-to-JSON Transformation [37].

JSONP-X: JSON envelope with XML content


Aside from offering JSON as a response format with callbacks, you can also specify XML as the response format. If in your query you specify a callback (callback=cbfunction) and also request the format be in XML (format=xml), then YQL returns a string representation of the XML within an array. Compare the following Yahoo! Local search for Indian restaurants in Sunnyvale, California using JSONP and JSONPX callbacks, respectively: JSONP Callback cfnto("ur"{cut:1""rae""090-00:32Z,ln""nU""pae""090-00:32Z, bucin{qey:"on""0,cetd:20-71T91:8""ag:e-S,udtd:20-71T91:8"

"r""tp/qeyyhopscmv/q?=eetTtefo+oa.erhweezp3%748%7adqey3%7ninrsarns2" ui:ht:/ur.aoai.o/1ylqslc+il+rmlclsac+hr+i%D29052+n+ur%D2ida+etuat%7, "diagnostics":{"publiclyCallable":"true", "url":{"execution-time":"332",

Yahoo! Developer Network

34

September 8, 2011

Response Data

"otn""tp/lclyhopscmLclerhevc/3lclerhzp905qeyida%0etuat&tr=&eut=0} cnet:ht:/oa.aoai.o/oaSacSrieV/oaSac?i=48&ur=nin2rsarnssat1rsls1",

"user-time":"335","service-time":"332","build-version":"2213"},"results":{"Result":[ {"Title":"Grand Indian Buffet"}, {"Title":"Turmeric Restaurant"}, {"Title":"Taj India"}, {"Title":"Shalimar"}, {"Title":"Komala Vilas"}, {"Title":"Brindavan Fine Indian Cuisine"}, {"Title":"Panchavati Indian Veggie Foods"}, {"Title":"Sneha Restaurant"}, {"Title":"Bhavika's Food to Go"}, {"Title":"ATHIDHI INDIAN CUISINE"}]}}}); JSONP-X Callback cfnto("ur"{cut:1""rae""090-00:02Z,ln""nU""pae""090-00:02Z, bucin{qey:"on""0,cetd:20-71T91:9""ag:e-S,udtd:20-71T91:9"

"r""tp/qeyyhopscmv/q?=eetTtefo+oa.erhweezp3%748%7adqey3%7ninrsarns2" ui:ht:/ur.aoai.o/1ylqslc+il+rmlclsac+hr+i%D29052+n+ur%D2ida+etuat%7, "diagnostics":{"publiclyCallable":"true", "url":{"execution-time":"558", "otn""tp/lclyhopscmLclerhevc/3lclerhzp905qeyida%0etuat&tr=&eut=0} cnet:ht:/oa.aoai.o/oaSacSrieV/oaSac?i=48&ur=nin2rsarnssat1rsls1",

"user-time":"561","service-time":"558","build-version":"2213"}},"results":[ "<Result xmlns=\"urn:yahoo:lcl\"><Title>Grand Indian Buffet<\/Title><\/Result>", "<Result xmlns=\"urn:yahoo:lcl\"><Title>Turmeric Restaurant<\/Title><\/Result>", "<Result xmlns=\"urn:yahoo:lcl\"><Title>Taj India<\/Title><\/Result>", "<Result xmlns=\"urn:yahoo:lcl\"><Title>Shalimar<\/Title><\/Result>", "<Result xmlns=\"urn:yahoo:lcl\"><Title>Komala Vilas<\/Title><\/Result>", "<Result xmlns=\"urn:yahoo:lcl\"><Title>Brindavan Fine Indian Cuisine<\/Title><\/Result>", "<Result xmlns=\"urn:yahoo:lcl\"><Title>Panchavati Indian Veggie Foods<\/Title><\/Result>", "<Result xmlns=\"urn:yahoo:lcl\"><Title>Sneha Restaurant<\/Title><\/Result>", "<Result xmlns=\"urn:yahoo:lcl\"><Title>Bhavika's Food to Go<\/Title><\/Result>", "<Result xmlns=\"urn:yahoo:lcl\"><Title>ATHIDHI INDIAN CUISINE<\/Title><\/Result>"]});

Yahoo! Developer Network

35

September 8, 2011

Response Data

Structure of Response
Every response from YQL includes a query element, which contains the diagnostics and results elements. (For details on the diagnostics element, see Information About the YQL Call [36].) The repeating elements within result are "rows" from a YQL table. For example, select * from social.connections returns multiple connection elements within the result element. The following listing shows the basic structure of the XML data in the response of a call to the YQL Web Service.

<query ... (attributes such as count)> <diagnostics> ... (sub-elements such as publiclyCallable) <results> ... (data returned by the call to YQL) </results> </query>

The next listing shows the basic structure of YQL response in JSON format:

{ "query": { "count": ... ... "diagnostics": { "publiclyCallable": ..., ... }, "results": { // data returned by call to YQL ... }

Information about the YQL Call


To get information about the execution of the YQL call, check the attributes of the query element and the sub-elements of the diagnostics element. The following table lists the attributes of the query element in an XML response. In a JSON response, these attributes are mapped to the name-value pairs contained in the query object. Attribute of query Element Description count created The number of items (rows) in returned by the YQL statement. In an XML response, count is the number of sub-elements in the results element. The date and time the response was created.

Yahoo! Developer Network

36

September 8, 2011

Response Data

Attribute of query Element Description lang updated The locale for the response. The date and time this response was last updated.

The diagnostics element contains information about the calls the YQL Web service made to backend datasources. The following table lists the XML sub-elements of the diagnostics element. In a JSON response, these sub-elements are mapped to name-value pairs contained in the diagnostics object. Sub-element of diagnostics Description Element publiclyCallable url True if the table is public data, false for private data. Authorization is required for private data. The URL of the Web service called by YQL to get the data. The value of the execution-time attribute is elapsed time, in milliseconds required to call the URL. The time YQL would take if each request were sequentially performed. YQL performs most request-related actions in parallel, so this measurement is informational only. The time YQL takes to perform the request and return the result, including time taken to execute Javascript. This is the time that you wait for each YQL statement response.

user-time

service-time

XML-to-JSON Transformation
If the YQL results are in JSON format, and the table is backed by an XML data source, then YQL transforms the data from XML to JSON. This transformation is "lossy," that is, you cannot transform the JSON back to XML. YQL transforms XML data to JSON according to the following rules: Attributes are mapped to name:value pairs. Element CDATA or text sections are mapped to "content":value pairs if the element contains attributes or sub-elements. Otherwise they are mapped to the element name's value directly. Namespace prefixes are removed from names. If the attribute, element, or namespace-less element would result in the same key name in the JSON structure, an array is created instead. For example, consider the following XML: <doc yahoo:count=10> <ns:a>avalue</ns:a> <b><subb>bvalue</subb></b> <c count=20 yahoo:count=30> <count>40</count> <count><subcount>10</subcount></count> </c> <d att="cat">dog</d> </doc> This XML is transformed to the following JSON structure:

Yahoo! Developer Network

37

September 8, 2011

Response Data

{doc: { count:10, a:"avalue", b: { subb: "bvalue"}, c: { count: [ 20,30,40,{subcount:10} ] }, d: { att:"cat", content:"dog" } }}

JSON-to-JSON Transformation
YQL transforms all JSON data sources into XML before returning results. To return JSON results that were obtained from a JSON data source, YQL must do the following: 1. transform the original JSON data to XML, 2. transform the XML back into JSON result. During the tranformation from XML to JSON, the original JSON may be altered or become "lossy". In other words, the original JSON may not be the same as the returned JSON. The original JSON may be altered in the following ways: JSON numbers are returned as strings. JSON arrays containing a single element are returned as a JSON object. To prevent this "lossy" transformation, you append the query string parameter jsonCompat=new [3] to the YQL Web Service URL that you are using. To illustrate how the jsonCompat parameter is used, we'll look at the below examples that use the community table that queries the Gowalla API, which only returns JSON. Lossy JSON The following REST URI uses the public YQL Web Service URL and the Gowalla table. The jsonCompat parameter is not added to the URI, so the original JSON returned from the Gowalla API will be altered in the YQL response. http://query.yahooapis.com/v1/public/yql?q=select * from gowalla.users where id='sco' and api_key='fa574894bddc43aa96c556eb457b4009'&env=store://datatables.org/alltableswithkeys In the returned "lossy" JSON results below, notice that last_checkins is an object and that _comments_count property has the the string value "0". ... "last_checkins": { "type": "checkin", "url": "/checkins/39776909", "spot": { "image_url": "http://static.gowalla.com/categories/190-2a44f344d6504e3b4510998e7c10f9bd-100.png", "url": "/spots/1440379", "name": "Authentic Smiles" }, "message": "<Mumble mumble>",

Yahoo! Developer Network

38

September 8, 2011

Response Data

"created_at": "2011-06-29T14:10:43Z", "_comments_count": "0" }, ... Lossless JSON Appending the jsonCompat=new query parameter to the REST URI as seen below, YQL now returns the same JSON data as the Gowalla API. http://query.yahooapis.com/v1/public/yql?q=select * from gowalla.users where id='sco' and api_key='fa574894bddc43aa96c556eb457b4009'&env=store://datatables.org/alltableswithkeys&jsonCompat=new In the returned "lossless" JSON results below, the last_checkins property is now an array with a single element, and _comments_count is a number. ... "last_checkins": [ { "type": "checkin", "spot": { "image_url": "http://static.gowalla.com/categories/190-2a44f344d6504e3b4510998e7c10f9bd-100.png", "url": "/spots/1440379", "name": "Authentic Smiles" }, "message": "<Mumble mumble>", "_comments_count": 0, "url": "/checkins/39776909", "created_at": "2011-06-29T14:10:43Z" } ], ...

Errors and HTTP Response Codes


The YQL Web Service returns the following HTTP response codes: Error 200 OK Description The YQL statement executed successfully. If the YQL statement is syntactically correct and if authorization succeeds, it returns 200 OK even if the calls to back-end data services return 400 or 500 errors. Information about these backend errors is in the diagnostics element of the response from YQL. Malformed syntax or bad query. This error occurs if the WHERE clause does not include a required input key. The XML error element includes a text description of the error. In the YQL Console, the error description appears in a highlighted bar.

400 Bad Request

Yahoo! Developer Network

39

September 8, 2011

Response Data

Error 401 Authorization quired

Description Re- The user running the application calling YQL is not authorized to access private data. This error also occurs if the user attempts to access his or her own private data without logging in to Yahoo!.

999 - Unable to process This error normally occurs when too many requests are sent to YQL and the this request at this time rate limit has been reached.

Yahoo! Developer Network

40

September 8, 2011

Chapter 6. Using YQL Open Data Tables


In this Chapter: Overview of Open Data Tables [41] Invoking an Open Data Table Definition within YQL [41] Setting Key Values for Open Data Tables [44]

Overview of Open Data Tables


YQL contains an extensive list of built-in tables for you to use that cover a wide range of Yahoo! Web services and access to off-network data. Open Data Tables in YQL allow you to create and use your own table definitions, enabling YQL to bind to any data source through the SQL-like syntax and fetch data. Once created anyone can use these definitions in YQL. An Open Data Table definition is an XML file that contains information as you define it, including, but not limited to the following: Authentication and Security Options: The kind of authentication you require for requests coming into your service. Also, whether you require incoming connections to YQL be made over a secure socket layer (via HTTPS). Sample Query: A sample query that developers can run via YQL to get information back from this connection. YQL Data Structure: Instructions on how YQL should create URLs that access the data available from your Web service. Also, an Open Data Table definition provides YQL with the URL location of your Web service along with the individual query parameters (keys) available to YQL. Pagination Options: How YQL should "page" through results. If your service can provide staggered results, paging will allow YQL to limit the amount of data returned.

Invoking an Open Data Table Definition within YQL


If you want to access external data that is not provided through the standard YQL set of tables (accessible through the show tables query), YQL provides the use statement when you want to import external tables defined through your Open Data Table definition.

Invoking a Single Open Data Table


You can access a single Open Data Table using the USE and AS verbs: USE "http://myserver.com/mytables.xml" AS mytable; SELECT * FROM mytable WHERE...

Yahoo! Developer Network

41

September 8, 2011

Using YQL Open Data Tables

Tip
The AS verb in the above example is optional. If you omit the AS verb, YQL uses the filename (without the .xml file ending) to name the table. In the above query, USE precedes the location of the Open Data Table definition, which is then followed by AS and the table as defined within your Open Data Table definition. After the semicolon, the query is formed as would be any other YQL query. YQL fetches the URL above and makes it available as a table named mytable in the current request scope. The statements following use can then select or describe the particular table using the name mytable.

Invoking Multiple Open Data Tables


You can also specify multiple Open Data Tables by using multiple USE statements in the following manner: USE "http://myserver.com/mytables1.xml" as table1; USE "http://myserver.com/mytables2.xml" as table2; SELECT * FROM table1 WHERE id IN (select id FROM table2)

Invoking Multiple Open Data Tables as an Environment


An easier way to use multiple Open Data Tables is to write or use a YQL environment file, which allows you to use multiple tables at once without the USE verb in your YQL statements. An environment file is simply a text file that contains a list of USE and SET statements, typically ending with a ".env" suffix. Here is how an environment file can look: USE 'http://www.datatables.org/amazon/amazon.ecs.xml' AS amazon.ecs; USE 'http://www.datatables.org/bitly/bit.ly.shorten.xml' AS bit.ly.shorten; USE 'http://www.datatables.org/delicious/delicious.feeds.popular.xml' AS delicious.feeds.popular; USE 'http://www.datatables.org/delicious/delicious.feeds.xml' AS delicious.feeds; USE 'http://www.datatables.org/dopplr/dopplr.auth.xml' AS dopplr.auth; USE 'http://www.datatables.org/dopplr/dopplr.city.info.xml' AS dopplr.city.info; USE 'http://www.datatables.org/dopplr/dopplr.futuretrips.info.xml' AS dopplr.futuretrips.info; USE 'http://www.datatables.org/dopplr/dopplr.traveller.fellows.xml' AS dopplr.traveller.fellows;

Tip
The AS verb in the above example is optional. If you omit the AS verb, YQL uses the filename (without the .xml file ending) to name the table. Once you upload the environment file to your server, you can simply access the YQL console and append the location of the file as follows: http://developer.yahoo.com/yql/console/?env=http://datatables.org/alltables.env

Yahoo! Developer Network

42

September 8, 2011

Using YQL Open Data Tables Try this example in the YQL console1

Tip
You can include multiple environment files at once by using multiple env query parameters. These are loaded in the order they appear in the query string. http://developer.yahoo.com/yql/console/?env=http://datatables.org/alltables.env &env=http://website.com/mytable.env

Working with Nested Environment Files


Advanced YQL users can better organize their Open Data Tables by nesting environment files within other environment files. For example, you can group similar tables into different environments and include each environment in a main (root) environment file as necessary. In addition to invoking environments, you can apply any SET statements to environments you invoke. The following example both invokes multiple environments within a root environment file, // Include the search group of tables env 'http://foo.com/searchgroup.env'; // Also use this api-key on all of these envs set apikey='abcsecret' on search; // Also include the Flickr group since my env requires them env 'http://foo.com/flickr.env;

Figure 6.1. Environment File Transversal

Usage Rules
YQL processes environment files using an in-order traversal. In the example diagram above: (A) is the root environment file. (B) and (D) are environments that are invoked by (A).

http://developer.yahoo.com/yql/console/?env=http://datatables.org/alltables.env

Yahoo! Developer Network

43

September 8, 2011

Using YQL Open Data Tables

(C) and (E) are environments that are invoked by (E). Tables are added in the following order (C), (B), (E), (D), (A) assuming that all tables are defined after the environment inclusions. The order in which you define tables can be changed by first using USE statements and then including other environments. SET statements can be called only on tables defined in either the defining environment or any of it parents. For example, in the example above, sets in (A)/(B)/(C) can apply to tables defined in (C). YQL detects and disallows circular references in environments inclusions. The last table definition for a given name takes precedence in a given environment. SET statements do not work across environments or across peers. For example, in the above example, (D) cannot set a key on tables defined in (B) or (C). If the same environment is included again as a peer, the last inclusions wins and overrides all previously SET statements. SET statements are scoped and the farthest parent with the most appropriate table prefix match wins.

Setting Key Values for Open Data Tables


For greater convenience and security, YQL allows you to set up key values for use within Open Data Tables. You can set values, such as passwords, API keys, and other required values, independently of YQL statements and API calls. The following example sets the api_key value within the YQL statement itself: select * from guardian.content.search where api_key="1234567890" and query="environment" The SET keyword allows you to set key values outside of a YQL statement, including environment files [42]. The SET keyword uses the following syntax within an environment file [42]: SET api_key="1234567890" ON guardian; In the example above, SET is followed by the key (api_key) and its value (1234567890). You must also specify the prefix of the table, which in this case is guardian. Once you set the key value within an environment file, remove these values in the YQL statement: select * from guardian.content.search where query="environment" Because key values set in an environment file using the SET keyword are not limited to a specific table binding, those keys must be defined as optional in the select binding [3] to prevent them from being used as local filters in SELECT statements. For example, suppose you want to set the game_status value for INSERT statements with the following: SET game_status="Available to play" ON online_games;

Yahoo! Developer Network

44

September 8, 2011

Using YQL Open Data Tables

Although you only intended to use game_status for INSERT statements, the key would still be used as a local filter in a SELECT statement unless it was declared as an optional key (with its required attribute set to "false") in the select binding, as shown here: ... <bindings> <select itemPath="online_games.player.updates" produces="XML"> ... <inputs> <key id="game_status" type="xs:string" paramType="variable" required="false" /> ... </inputs> ... </select> </binding> ... The following precedence rules apply when setting key values with the SET keyword: Keys that are set within the YQL statement take precedence over keys that are set using the SET keyword. If the set key is defined more than once, the most precise definition, based on the length of the table prefix, takes precedence. If the set key is defined more than once with the same preciseness, the last definition is used.

Using SET to Hide Key Values or Data


To avoid exposing private data when you share YQL Open Data Tables, you can use a combination of YQL features to hide such data: 1. Add your private values to an environment file using the SET keyword. 2. Use the yql.storage.admin table to import the environment file or an Open Data Table with a memorable name. YQL provides you with a set of shared access keys. 3. Use the shared execute or select access keys [72] as you would an Open Data Table, environment file, or JavaScript.

Important
Ensure that you place all USE and SET statements together respectively in one environment file to prevent private data being handed over to redefined tables.

Yahoo! Developer Network

45

September 8, 2011

Creating YQL Open Data Tables

Creating YQL Open Data Tables


Abstract This part of the YQL Guide provides a reference for YQL Open Data Tables. It also discusses using server-side JavaScript and hosted storage to extend the abilities of Open Data Tables. Before reading this part of the YQL Guide, you should first read Introducing YQL [1] and Using YQL and Open Data Tables [1].

Table of Contents
1. Creating YQL Open Data Tables ......................................................................................... 1 Before You Begin ........................................................................................................ 1 Open Data Tables Reference .......................................................................................... 1 table element ...................................................................................................... 2 meta element ...................................................................................................... 2 select / insert / update / delete elements .................................................................... 3 function Element ............................................................................................... 37 url element ......................................................................................................... 5 execute element ................................................................................................... 5 key / value / map elements ..................................................................................... 6 paging element .................................................................................................... 9 YQL Streaming ......................................................................................................... 11 Configuring Open Data Tables to Use Streaming ...................................................... 11 Using Open Data Tables Configured for Streaming ................................................... 12 Debugging Open Data Tables and YQL Network Calls ...................................................... 12 Enabling Logging .............................................................................................. 13 Viewing Logs .................................................................................................... 13 Open Data Table Examples .......................................................................................... 14 Flickr Photo Search ............................................................................................ 14 Digg Events via Gnip .......................................................................................... 15 Open Data Tables Security and Access Control ................................................................ 16 Batching Multiple Calls into a Single Request ................................................................. 17 Troubleshooting ......................................................................................................... 17 2. Executing JavaScript in Open Data Tables ........................................................................... 19 Introduction .............................................................................................................. 19 Features and Benefits .......................................................................................... 19 Ensuring the Security of Private Information ................................................................... 20 JavaScript Objects, Methods, and Variables Reference ....................................................... 20 y Global Object ................................................................................................. 21 request Global Object ......................................................................................... 30 rest Object ........................................................................................................ 30 response Global Object ....................................................................................... 37 result Object ..................................................................................................... 37 internal Global Object ......................................................................................... 38 Global Variables ................................................................................................ 42 JavaScript and E4X Best Practices for YQL .................................................................... 43 Paging Results ................................................................................................... 43 Including Useful JavaScript Libraries ..................................................................... 43 Using E4X within YQL ....................................................................................... 44 JavaScript Logging and Debugging ....................................................................... 46 Executing JavaScript Globally .............................................................................. 46 Making Asynchronous Calls with JavaScript Execute ................................................ 47 Preventing Lossy JSON Results ............................................................................ 50 Examples of Open Data Tables with JavaScript ................................................................ 50 Hello World Table .............................................................................................. 51 Yahoo! Messenger Status ..................................................................................... 51 OAuth Signed Request to Netflix .......................................................................... 52 Request for a Flickr frob ...................................................................................... 54 Celebrity Birthday Search using IMDB .................................................................. 55 Shared Yahoo! Applications ................................................................................. 58 CSS Selector for HTML ...................................................................................... 60

Yahoo! Developer Network

iii

September 8, 2011

Creating YQL Open Data Tables

Execution Rate Limits ................................................................................................. 3. Creating YQL Customized Functions .................................................................................. Introduction .............................................................................................................. Types of Functions ..................................................................................................... How to Use a Customized Function ............................................................................... function Element ....................................................................................................... Stream Functions ....................................................................................................... Example Stream Function .................................................................................... Reduce Functions ....................................................................................................... Reference ................................................................................................................. 4. Using Hosted Storage with YQL ........................................................................................ Introduction .............................................................................................................. About YQL Hosted Storage ................................................................................. Storage Limits and Requirements .......................................................................... Storing New Records .................................................................................................. Storing a New Record using Text .......................................................................... Storing a New Record using Data from an URL ....................................................... Storing a New Named Record using Data from an URL ............................................ Using YQL to Read, Update, and Delete Records ............................................................. Accessing Records using YQL .............................................................................. Deleting Records using YQL ................................................................................ Updating Records using YQL ............................................................................... Using Records within YQL .......................................................................................... Using Hosted Environment Files ........................................................................... Using Hosted YQL Open Data Tables .................................................................... Including Hosted JavaScript .................................................................................

61 63 63 63 63 64 64 65 66 68 69 69 69 69 70 70 71 71 71 71 72 72 72 73 73 73

Yahoo! Developer Network

iv

September 8, 2011

Chapter 1. Creating YQL Open Data Tables


In this Chapter: Open Data Tables Reference [1] Debugging Open Data Tables and YQL Network Calls [12] Open Data Table Examples [14] Open Data Tables Security and Access Control [16] Batching Multiple Calls into a Single Request [17] Troubleshooting [17]

Before You Begin


Before reading this chapter, you should first read Introducing YQL [1] and Using YQL and Open Data Tables [1]

Open Data Tables Reference


The following reference describes the structure of an Open Data Table definition: The following elements and sub-elements of YQL Open Data Tables are discussed in this reference: tables [2] meta [2] bindings select / insert/ update / delete [3] urls url [5] inputs key / value / map [6] paging [9] pagesize [10] start [10] total [10] nextpage [11]

Yahoo! Developer Network

September 8, 2011

Creating YQL Open Data Tables

table element
Full Path: root element Example: <table xmlns="http://query.yahooapis.com/v1/schema/table.xsd"> This is the root element for the document. A table is the level at which an end-user can 'select' information from YQL sources. A table can have many different bindings or ways of retrieving the data. However, we advise that a single table produce a single type of result data. Attribute xmlns URL Value(s) Notes The XML Schema file related to this Open Data Table definition.

s e c u r i t y - enumeration, any / The authorization level required to access. Level app / user any: Anonymous access; any user can access this table. app: 2-legged OAuth; involves authorization without access to private user data. user: 3-legged OAuth, involves authorization of access to user data. For more information, refer to Open Data Tables Security and Access Control [16]. https boolean, true false or If true, the table is only available if the user is connected via HTTPS. If missing or false, either HTTP or HTTPS connections are acceptable.

Warning
If your table requires input that is deemed "private", such as any passwords, authentication keys, or other "secrets", you MUST ensure the https attribute within the table element is set to true.

meta element
Full Path: table/meta Example: <meta> <author>Yahoo! Inc.</author> <documentationURL>http://www.flickr.com/services/api/flickr.photos.search.html</documentationURL> <sampleQuery>select * from {table} where has_geo="true" and text="san francisco"</sampleQuery> </meta> Along with the table element, you are required to include the meta sub-element, which provides the following information:

Yahoo! Developer Network

September 8, 2011

Creating YQL Open Data Tables

Attribute sampleQuery

Description

Notes

A sample query that users can run {table} should be used in place of the table to get output from this table. name, so that the sample can run when used in different namespaces. Multiple sampleQuery elements may occur. Each sampleQuery may have a description attribute that contains a description about the sample.

d o c u m e n t a - Additional information about this More then one documentationURL element table or the select called by the table may be included for each table. tionURL can be found here description author Plain text description about the table A description of the table. Information regarding the author of Examples of author information include an unthis Web service formatted email, name, or other related information.

select / insert / update / delete elements


Full Path: table/bindings/select table/bindings/insert table/bindings/update table/bindings/delete Example: <bindings> <select itemPath="rsp.photos.photo" produces="XML"> ... </bindings> Situated within each bindings element, there are one of four keywords: select, insert, update, or delete. The select element describes the information needed for YQL to read data from an API. The insert and update elements describe the information needed to add or modify data from an API, respectively. When removing data, the delete element is used to describe the necessary bindings. When a keyword such as select or update is repeated within the bindings array, it can be considered to be an alternative way for YQL to call a remote server to get the same type of structured data. Typically, this is needed when the service supports different sets of query parameters (YQL's "keys") or combinations of optional query parameters. Attribute itemPath URLs Value(s) Notes A dot-path that points to where the repeating data elements occur in the response format. These are the "rows" of your table.

Yahoo! Developer Network

September 8, 2011

Creating YQL Open Data Tables

Attribute

Value(s)

Notes The frequency in seconds that the YQL Engine should update the response from a data source. See YQL Streaming for more information.

p o l l i n g F r e - Integer quencySeconds produces

enumeration, XML / The type of data coming back from the Web service. JSON

Note
Unlike XML, JSON objects have no "root" node. To work with the dot notation, YQL creates a "pseudo" root node for JSON responses called "json". If you need to return a sub-structure from your Open Data Table that fetches or produces JSON, you'll need to add "json" at the root of the path.

function Element
A stored procedure to bind and run, similar to the built-in Sort and Other Functions1. For more information on the function element, see Creating YQL Customized Functions [63]. . Full Path: table/bindings/function Example: <table> <bindings> <function name="concat" /> <inputs> ... </inputs> <execute> ... </execute> </function> ... </bindings> <table> Attribute Type name type string Description The name of the function.

enumeration The type of function. Allowed values: stream - (default) This type of function gets called for each item at which point it can modify and return the modified version of the item. Example: Decorate an item with extra information. reduce - This type of function gets called for each item which is collected/stored and then finally, gets one last call to return the reduced or aggregated response. Examples of YQL built-in reduce functions: sort, truncate, count.

http://developer.yahoo.com/yql/guide/sorting.html

Yahoo! Developer Network

September 8, 2011

Creating YQL Open Data Tables

url element
Full Path: table/bindings/select/urls/url This is where YQL and the table supporting the service come together. The url element describes the URL that needs to be executed to get data for this table, given the keys in the key elements.

Note
The CDATA/TEXT for this element contains the URL itself that utilizes substitution of values at runtime based on the uri template spec (v3)2. The names of the values will be substituted and formatted according to the uri template spec, but the simplest method is simply to enclose a key name with curly braces ( {} ): All {name}keys found in the URL will be replaced by the same id key value in the keys elements. YQL currently supports both http and https protocols. Example: https://prod.gnipcentral.com/publishers/{publisher}/notification/{bucket}.xml YQL will look for key elements with the names publisher and bucket. If the YQL developer does not provide those keys in the WHERE clause (and they are not optional), then YQL detects the problem and will produce an error. If an optional variable is not provided, but is part of the Open Data Table definition, it will be replaced with an empty string. Otherwise, YQL will substitute the values directly into the URL before executing it.

execute element
Full Path:table/bindings/select/execute The execute sub-element allows you to invoke server-side JavaScript in place of a GET request. For more information on executing JavaScript, refer to Executing JavaScript within Open Data Tables [19]. Example: <execute> <![CDATA[ // Include the flickr signing library y.include("http://blog.pipes.yahoo.net/wp-content/uploads/flickr.js"); // GET the flickr result using a signed url var fs = new flickrSigner(api_key,secret); response.object = y.rest(fs.createUrl({method:method, format:""})).get().response();

http://tools.ietf.org/html/draft-gregorio-uritemplate-03

Yahoo! Developer Network

September 8, 2011

Creating YQL Open Data Tables

]]> </execute>

key / value / map elements


Full Paths: table/bindings/[select/insert/update/delete]/inputs/key table/bindings/[select/insert/update]/inputs/value table/bindings/[select/insert/update/delete]/inputs/map Example: <inputs> <key id='guid' type='xs:string' paramType='path' required="true" /> <key id='ck' type='xs:string' paramType='variable' required="true" /> <key id='cks' type='xs:string' paramType='variable' required="true" /> <value id='content' type='xs:string' paramType='variable' required="true" /> </inputs> There are three type of elements available within the inputs element: key, value, and map.

key element
Each key element represents a named "key" that you provide in the WHERE or INTO clause of SELECT, INSERT, UPDATE, or DELETE statements. YQL inserts these values into the URL request before it is sent to the server. YQL inserts these values into the URL request if the paramType is set to query or path or header. For a variable type, the key named as the id of the element is made available in the execute section of the Open Data Table.

value element
Use the value element to assign a new "value" or update an existing one within an Open Data Table. The value element defines a field that can only be set as an input and therefore cannot be in YQL statements to satify the "where" clause. The value element only works with the INSERT and UPDATE verbs and in different ways. When used with the insert keyword, the value element appears in the VALUE expression of the YQL statement, indicating that a new value is being passed into the YQL statement, as seen in the following example: INSERT into bitly.shorten (login, apiKey, longUrl) VALUES ('YOUR_LOGIN', 'YOUR_API_KEY', 'http://yahoo.com') When used with the update keyword, the value element is called from the SET portion of the YQL statement. This indicates that you are "setting" a particular value, as seen in the following example: UPDATE table SET status='Reading the YQL Guide' where guid = me;

Yahoo! Developer Network

September 8, 2011

Creating YQL Open Data Tables

map element
Use the map element when you want to use dynamic keys. With this element, YQL uses the value you pass in through the YQL statement as a variable. This variable is used within the execute portion of your Open Data Table to determine what action to take. For example, you may set up a YQL Open Data Table that updates either bit.ly, delicio.us, or tinyurl, depending on the value you specify in the YQL statement. For a dynamic key called type, the actual ID in a YQL query would look like the following: field.type = 'Java'

Note
In the absence of the map element as a binding, all identifiers, not corresponding to a binding element and that appear in a YQL query, are treated as local filters. The map element can be used for all the four paramTypes [9]. Here is an example of the map element being used in a path: <map id="field" paramType="path"/> For a query containing the relational expression field.type='rss', only the dynamic parameter name type would be substituted in the urls element. The URI template would look like the following: http://rss.news.yahoo.com/{type}/topstories

Key, Value, and Map Element Support within YQL Statements


The following table shows the keywords that support the key, value, and map elements: select insert update delete key map yes yes yes yes yes yes yes yes yes no yes value no

Attributes for Key, Value, and Map Elements


The following table provides the attributes available within key, value and map elements: Attribute id as Value(s) string string Supported Keywords Notes

select, insert, up- The name of the key. This represents what the user date, delete needs to provide in the WHERE clause. select, insert, up- The alias of the key used in YQL statements. date, delete If the Web source used in the Open Data Table uses a cryptic or poorly named query parameter, you can use as to specify an alias that developers use in the YQL statement. For example, perhaps you have an id called "q" within your Open Data Table, which actually is a search parameter.

Yahoo! Developer Network

September 8, 2011

Creating YQL Open Data Tables

Attribute

Value(s)

Supported Keywords

Notes Without aliasing, the equivalent YQL statement would look like this: select * from google.search where q = "pizza" You can use the as attribute to create an alias in the following way: <key id="q" as="query" type="xs:string" paramType="query"/> You then can use search in your YQL statement like this: select * from google.search where query = "pizza"

type required

string

select, insert, up- The type of data coming back from the Web service. date, delete

boolean select, insert, up- A boolean that answers the question: Is this key required date, delete to be provided in the WHERE clause on the left-hand side of an equality statement? If not set, any key is optional. enumera- select, insert, up- Determines how this key is represented and passed on tion date, delete to the Web service: - query: Add the id and its value as a id=value query string parameter to the URL. - matrix: Add the id and its value as a id=value matrix parameter to the URL path. - header: Add the id and its value as a id: value as an HTTP header to the URL request. - path: Substitute all occurrences of {id} in the url string with the value of the id. Not necessarily only in the path. - variable: Use this key or field as a variable to be used within the execute sub-element [19] instead of being used to format or form the URL.

paramType

default private

string

select, insert, up- This value is used if one isn't specified by the developer date, delete in the SELECT.

boolean select, insert, up- Hide this key's value to the user (in both "desc" and date, delete "diagnostics"). This is useful for parameters like appid and keys. boolean select, insert, up- A boolean that indicates whether the default attribute date, delete must be present and cannot be changed by the end user. Constant keys are not shown in desc [table].

const

Yahoo! Developer Network

September 8, 2011

Creating YQL Open Data Tables

Attribute batchable

Value(s)

Supported Keywords

Notes

boolean select, update, de- A boolean which answers the question: Does this select lete and URL support multiple key fetches/requests in a single request (batched fetching)? For more information about batching requests, refer to Batching Multiple Calls in a Single Request [17].

maxBatchItems integer

select, update, de- How many requests should be combined in a single lete batch call. For more information about batching requests, refer to Batching Multiple Calls in a Single Request [17].

Aliasing within Key, Value, and Map Elements


If you have an obscurely named id in your Open Data Table, you can use an alias to refer to it within YQL statements. For example, perhaps you have an id called "q" within your Open Data Table, which actually is a search parameter. You can use "as" to create an alias in the following way: <key id="q" as=type="xs:string" paramType="query"/> select * from google.search where search ="pizza"

paging element
Full Path: table/bindings/select/paging Examples: <paging model="page"> <start id="page" default="0" /> <pagesize id="per_page" max="250" /> <total default="10" /> </paging> <paging model="url"> <nextpage path="ysearchresponse.nextpage" /> </paging> <paging model="offset" matrix="true"> <start id="page" default="0" /> <pagesize id="per_page" max="250" /> <total default="10" /> </paging> This element describes how YQL should "page" through the web service results, if they span multiple pages, or the service supports offset and counts. Attribute model Value(s) Supported Keywords Notes The type of model to use to fetch more than the initial result set from the service.

o f f s e t , select page, url

Yahoo! Developer Network

September 8, 2011

Creating YQL Open Data Tables

Attribute

Value(s)

Supported Keywords

Notes The offset refers to services that allow arbitrary index offsets into the result set. Use the page value for services that support distinct "pages" of some number of results. Use the url value for services that support a URL to access further data.

matrix true, false

select

A boolean that answers the question: Is the parameter matrix style (part of the URI path; delimited), or query parameter style? The default value is false.

Tip
When using the url paging model, you can also use the pagesize element to, if the Web service allows, adjust the number of results returns at once.

pagesize element
Full Path: table/bindings/select/paging/pagesize This element contains Information about how the number of items per request can be specified. Attribute Value(s) max integer Notes The maximum size of the requested page. If the total requested is below the max pagesize, then the pagesize will be the total requested. Otherwise, the max pagesize will be the size of the page requested. The name of the parameter that controls this page size.

id

string

start element
Full Path: table/bindings/select/paging/start This element contains Information about how the "starting" item can be specified in the set of results. Attribute Value(s) default integer id string Notes The starting item number (generally 0 or 1); for paging style this value always defaults to 1. The name of the parameter that controls the starting page/offset.

total element
Full Path: table/bindings/select/paging/total This element contains Information about the total number of results available per request by default.

Yahoo! Developer Network

10

September 8, 2011

Creating YQL Open Data Tables

Attribute Value(s) default integer

Notes The number of items that come back by "default" in YQL if the () syntax is not used when querying the table.

nextpage element
Full Path: table/bindings/select/paging/nextpage This element contains the location of the next page of results. This is an optional element that is used in conjunction with the parent url element. Attribute Value(s) path string Notes The path to the next page of results

YQL Streaming
In general, when a YQL statement references an Open Data Table, the YQL Web service will make an HTTP call based on the URLs listed in the bindings element or execute the JavaScript in the execute element. To get updated results from the YQL Web service, you need to execute this YQL statement again. YQL streaming allows you to get updated results in a different way. With YQL streaming, your Open Data Table can request YQL to poll data from your resource URI at a specified interval, process the responses, and return the results to the client, without the client having to execute the same YQL statements over and over again to get the updated results; this will reduce network latency and the need to store responses.

Configuring Open Data Tables to Use Streaming


To request streaming in your Open Data Table, you use the pollingFrequencySeconds attribute of the select binding [7] to specify the interval that you want YQL to make requests. For example, the code snippet below configures YQL to poll the URL for Foursquare search every two seconds: <bindings> <select itemPath="" pollingFrequencySeconds="2"/> <urls> <url>https://api.foursquare.com/v2/users/search</url> </urls> <inputs> <key id="oauth_token" type="xs:string" paramType="query" required="true" /> <key id="phone" type="xs:string" paramType="query" /> <key id="email" type="xs:string" paramType="query" /> <key id="twitter" type="xs:string" paramType="query" /> <key id="twitterSource" type="xs:string" paramType="query" /> <key id="fbid" type="xs:string" paramType="query" /> <key id="name" type="xs:string" paramType="query" /> </inputs> </binding>

Yahoo! Developer Network

11

September 8, 2011

Creating YQL Open Data Tables

Note
When setting a polling frequency with pollingFrequencySeconds, try to match the update frequency of your data source. If you configure YQL to poll the source every two seconds, but the source usually updates data every minute, you will be increasing the network latency for YQL without the benefit of getting new data.

Using Open Data Tables Configured for Streaming


To use Open Data Tables that are configured for streaming, you must call the YQL Web Service URLs for streaming. You use YQL statements and invoke the Open Data Table in the same way as you would for any Open Data Table. The following YQL Service URL is used to get public data from an Open Data Table that is configured to use YQL streaming: http://query.yahooapis.com/v1/public/streaming/yql The YQL Service URL below requires authorization by OAuth and allows access to both public and private data from an Open Data Table that is configured to use YQL streaming: http://query.yahooapis.com/v1/streaming/yql The following PHP code snippet shows how to get public data from the Open Data Table books_warehouse.xml that is configured to use YQL streaming: // The YQL Service URL for streaming public data $YQL_STREAMING_URL = "https://query.yahooapis.com/v1/public/streaming/yql"; // Form YQL query and build URI to YQL Web service $yql_query = "use 'http://yql.lotsofbooks.com/books_warehouse.xml' as bw; select * from bw where isbn-10='9781849510707'"; $yql_query_url = $YQL_STREAMING_URL . "?q=" . urlencode($yql_query) . "&format=json"; // Make call with cURL and get returned response $session = curl_init($yql_query_url); curl_setopt($session, CURLOPT_RETURNTRANSFER,true); $json = curl_exec($session); // Convert JSON to PHP object $phpObj = json_decode($json);

Debugging Open Data Tables and YQL Network Calls


To aid in your debugging efforts, YQL provides the option to have network-level logging. When enabled, all network requests are uncached, so you can iteratively develop Open Data Tables more easily, as well as debug network requests between YQL and the remote service. Network logs display both the request headers and the response content for each network call.

Yahoo! Developer Network

12

September 8, 2011

Creating YQL Open Data Tables

Note
Network capture stops once YQL encounters a default (built-in) table or table fetched from yql.storage [69], even if subsequent requests are made using an Open Data Table. When you enable network-level logging, YQL provides a key within the diagnostics element for each network call that occurs, seen in the following YQL response snippet as id attributes: ... <diagnostics> <publiclyCallable>true</publiclyCallable> <url execution-time="11" id="5b81e4c4-11eb-43a5-866b-b1217498843e" proxy="DEFAULT"><![CDATA[http://datatables.org/alltables.env]]></url> <url execution-time="6" proxy="DEFAULT"><![CDATA[http://www.datatables.org/zillow/zillow.search.xml]]></url> <url execution-time="139" id="9dd99f2c-54e3-493f-a818-9950f0798d2a" poy"EAL"<[DT[tp/wwzlo.o/esrieGterheut.t?w-dX-W1s6itb1bvades13%03d2Ae2N&iyttzp909]<ul rx=DFUT>!CAAht:/w.ilwcmwbevc/eSacRslshmzsi=1Zzce8ac_3w&drs=8527r%0v%0Ectsaei=83]>/r> <user-time>183</user-time> <service-time>156</service-time> <build-version>2355</build-version> </diagnostics> ... The id key can be used within 5 minutes of an execution to see log data.

Enabling Logging
To enable network-level logging, you simply append debug=true to the YQL console URL or API query like this: http://query.yahooapis.com/v1/yql?q=select%20*%20from%20social.profile %20where%20guid%3Dme&format=xml&env=http%3A%2F%2Fdatatables.org%2Falltables.env&debug=true

Viewing Logs
You can access network-level logs within 5 minutes of running a YQL statement or call. Simply append the id key provided in diagnostics to the following URL: http://query.yahooapis.com/v1/logging/dump?id= Example: http://query.yahooapis.com/v1/logging/dump?id=5b81e4c4-11eb-43a5-866bb1217498843e

Yahoo! Developer Network

13

September 8, 2011

Creating YQL Open Data Tables

Open Data Table Examples


This section includes a few examples of Open Data Tables that showcase the ability of YQL to gather data from external APIs.

Tip
For a much larger list of publicly available Open Data Tables, refer to datatables.org3. Flickr Photo Search [14] Access to Digg Events using Gnip [15]

Flickr Photo Search


This Open Data Table definition ties into the Flickr API and allows YQL to retrieve data from a Flickr photo search: <?xml version="1.0" encoding="UTF-8"?> <table xlmns="http://query.yahooapis.com/v1/schema/table.xsd"> <meta> [2] <author>Yahoo! Inc.</author> <documentationURL>http://www.flickr.com/services/api/flickr.photos.search.html</documentationURL> <sampleQuery>select * from {table} where has_geo="true" and text="san francisco"</sampleQuery> </meta> [2] <bindings> <select itemPath="rsp.photos.photo" produces="XML"> <urls> <url env="all">http://api.flickr.com/services/rest/?method=flickr.photos.search</url> </urls> <paging model="page"> <start id="page" default="0" /> <pagesize id="per_page" max="250" /> <total default="10" /> </paging> <inputs> <key id="woe_id" type="xs:string" paramType="query" /> <key id="user_id" type="xs:string" paramType="query" /> <key id="tags" type="xs:string" paramType="query" /> <key id="tag_mode" type="xs:string" paramType="query" /> <key id="text" type="xs:string" paramType="query" /> <key id="min_upload_date" type="xs:string" paramType="query" /> <key id="max_upload_date" type="xs:string" paramType="query" /> <key id="min_taken_date" type="xs:string" paramType="query" />
3

http://datatables.org

Yahoo! Developer Network

14

September 8, 2011

Creating YQL Open Data Tables

<key <key <key <key <key <key <key <key <key />

id="max_taken_date" type="xs:string" paramType="query" /> id="license" type="xs:string" paramType="query" /> id="privacy_filter" type="xs:string" paramType="query" /> id="bbox" type="xs:string" paramType="query" /> id="accuracy" type="xs:string" paramType="query" /> id="safe_search" type="xs:string" paramType="query" /> id="content_type" type="xs:string" paramType="query" /> id="machine_tags" type="xs:string" paramType="query" /> id="machine_tag_mode" type="xs:string" paramType="query"

<key id="group_id" type="xs:string" paramType="query" /> <key id="contacts" type="xs:string" paramType="query" /> <key id="place_id" type="xs:string" paramType="query" /> <key id="media" type="xs:string" paramType="query" /> <key id="has_geo" type="xs:string" paramType="query" /> <key id="lat" type="xs:string" paramType="query" /> <key id="lon" type="xs:string" paramType="query" /> <key id="radius" type="xs:string" paramType="query" /> <key id="radius_units" type="xs:string" paramType="query" /> <key id="extras" type="xs:string" paramType="query" /> <key id="api_key" type="xs:string" const="true" private="true" paramType="query" default="45c53f8...d5f645"/> </inputs> </select> </bindings> </table> Run this example in the YQL console.4

Tip
To get a better understanding of how bindings work within YQL Open Data Tables, compare the Open Data Table definition above to photo.search on the Flickr API5.

Digg Events via Gnip


The following example ties into the Gnip API6 to retrieve activities from a Publisher, which in this case is Digg. <?xml version="1.0" encoding="UTF-8"?> <table xmlns="http://query.yahooapis.com/v1/schema/table.xsd"> <meta> <sampleQuery>select * from {table} where publisher='digg' and action='dugg'</sampleQuery> </meta> <bindings> <select itemPath="activities.activity" produces="XML" > <urls>
4

h t t p : / / d e v e l o p e r. y a h o o . c o m / y q l / c o n s o l e / ? q = s e l e c t % 2 0 * % 2 0 f r o m % 2 0 fl i c k r. p h o tos.search%20where%20has_geo%3D%22true%22%20and%20text%3D%22san%20francisco%22&env=http%3A%2F%2Fgithub.com%2Fspullara%2Fyql-tables%2Fraw%2Fef685688d649a7514ebd27722366b2918d966573%2Falltables.env 5 http://www.flickr.com/services/api/flickr.photos.search.html 6 http://docs.google.com/View?docid=dgkhvp8s_5svzn35fw#Examples_of_Activities

Yahoo! Developer Network

15

September 8, 2011

Creating YQL Open Data Tables

<url env="all">https://prod.gnipcentral.com/publishers/{publisher}/notification/{bucket}.xml</url> </urls> <inputs> <key id="publisher" type="xs:string" paramType="path" required="true" /> <key id="bucket" type="xs:string" paramType="path" required="true" /> <key id="Authorization" type="xs:string" paramType="header" const="true" default="Basic eXFsLXF1ZXN...BpcGVz" /> </inputs> </select> <select itemPath="activities.activity" produces="XML" useProxy="true" auth="callback"> <urls> <url env="all">https://prod.gnipcentral.com/publishers/{publisher}/notification/current.xml</url> </urls> <inputs> <key id="publisher" type="xs:string" paramType="path" required="true" /> <key id="Authorization" type="xs:string" paramType="header" const="true" default="Basic eXFsLXF1ZXN0a...BpcGVz" /> </inputs> </select> </bindings> </table> Run this example in the YQL console.7

Open Data Tables Security and Access Control


The securityLevel attribute of the table element determines the type of authentication required to establish a connection. In order for a user to connect to your table, the user must be authorized at the level or higher than the level indicated in the securityLevel attribute. The following table lists whether access is available depending on the value in the securityLevel attribute. Security of Table (access at- Anonymous / No Authoriz- 2-legged OAuth 3-legged OAuth / cookie ation tribute) any app user yes no no yes yes no yes yes yes

For more information about each level, refer to the securityLevel attribute in the table element [2].

http://developer.yahoo.com/yql/console/?q=select%20*%20from%20gnip.activity%20where%20publisher%3D%27digg%27%20and%20action%3D%27dugg%27

Yahoo! Developer Network

16

September 8, 2011

Creating YQL Open Data Tables

Batching Multiple Calls into a Single Request


YQL Open Data Tables support the ability to send a list of keys in a single request, batching up what would otherwise be a multiple calls.

Note
In order to do batching in YQL Open Data Tables, the source must support it. An example of a source that supports batching is the Yahoo! Social Directory call for profiles8) Let's take the example of the Social Directory API and see the URI for profile data: http://social.yahooapis.com/v1/user/{guid}/profile In YQL, the table for retrieving this data is social.profile. Here is an example that includes a subselect: select * from social.profile where guid in (select guid from social.connections where owner_guid = me) When performing sub-selects, the inner sub-select returns a set of values, each of which is a call to the URI above. So if a Yahoo! user has 3 connections in his profile, the sub-select makes three calls to the Social Directory API: http://social.yahooapis.com/v1/user/1/profile http://social.yahooapis.com/v1/user/2/profile http://social.yahooapis.com/v1/user/3/profile Fortunately, the Social Directory URI above also supports batching, so a single call can be made to get all three profiles: http://social.yahooapis.com/v1/users.guid(1,2,3)/profile Since the Social Directory API supports batching, YQL can enable this by defining the key guid as batchable [6] with an extra parameter that denotes the max number of batch items per request: <key id="guid" type="xs:string" paramType="path" batchable="true" maxBatchItems="3"/> We also need to modify the Open Table definition to support multiple values for the GUID. Combining the modification to the Open Table definition above with the one below results in a batch call to the Social Directory API for not more than 3 profiles: <url env="int">http://socialstuff.com/v1/users.guid({-listjoin|,|guid})/profile</url>

Troubleshooting
The following section deals with issues you may have while using YQL Open Data Tables: Problem: The SELECT URL doesn't parse correctly in my Open Data Table definition.
8

http://developer.yahoo.com/social/rest_api_guide/extended-profile-resource.html

Yahoo! Developer Network

17

September 8, 2011

Creating YQL Open Data Tables

Solution: Make sure you've escaped things correctly for XML; for example: & should be encoded as &amp;. Problem: My Open Data Table definition has multiple bindings with different sets of keys. YQL keeps running the "wrong" select. How can I get YQL to choose the right one? Solution: Keep in mind that the order of bindings is important. Once YQL finds a select that satisfies the YQL statement, it uses that one. Try moving the more "specific" select endpoint above the others. Problem: If my API requires authentication, how do I access it? Solution: If you want to use an API that requires its own authentication mechanism, you use the execute [19] sub-element within an Open Data Table to manage this authentication. Problem: Open Data Tables seem so complicated? What is the best way to get started? Solution: The best way to avoid being overwhelmed is to first look at examples [5]. In general, when creating YQL tables, it is useful to take a bottom-up approach and analyze the result structure of the API(s) that you are encapsulating. First, group together all the services that produce the same result structure. This becomes your "table" or Open Table definition. For each API that produces the response structure, you should create a "select" under the "request" section of the Open Data Table definition. By using this mechanism, you can often consolidate multiple API's into a single versatile YQL table that allows YQL to do the heavy lifting and keep the implementation details hidden.

Yahoo! Developer Network

18

September 8, 2011

Chapter 2. Executing JavaScript in Open Data Tables


In this Chapter: Introduction [19] Ensuring the Security of Private Information [20] JavaScript Objects, Methods, and Variables Reference [20] JavaScript and E4X Best Practices for YQL [43] Examples of Open Data Tables with JavaScript [50] Execution Rate Limits [61]

Introduction
Features and Benefits
The ability to execute JavaScript extends the functionality of Open Data Tables [41] in many ways, including the following: Flexibility beyond the normal templating within Open Data Tables: Executing JavaScript allows you to use conditional logic and to format data in a granular manner. Better data shaping and parsing: Using JavaScript, you can take requests and responses and format or shape them in way that is suitable to be returned. Better support for calling external Web services: Some Web services use their own security and authentication mechanisms. Some also require authentication headers to be set in the Web service request. The execute element allows you to do both. Better support for adding, modifying, and deleting data using external Web services: For Web services that support write access, YQL allows you to insert, update, and delete using server-side JavaScript within the insert, update, and delete elements, which are nested within the binding element. Ability to make asynchronous calls: YQL provides JavaScript methods for making REST calls that take a callback function parameter to handle the returned response. Using these methods with callback functions, you can initiate multiple calls that won't be blocked by a slow response or a timed-out request. The ability to execute JavaScript is implemented through the execute sub-element within an Open Data Table definition. Within the execute sub-element, you can embed JavaScript and E4X (the shortened term for ECMAScript for XML), which adds native XML support to JavaScript. Support for E4X was first introduced in JavaScript 1.6. When a YQL statement calls an Open Table Definition that contains the execute sub-element, YQL no longer performs the request to the templated URI in the endpoint. Instead YQL provides a runtime envir-

Yahoo! Developer Network

19

September 8, 2011

Executing JavaScript in Open Data Tables onment in which the JavaScript is executed server-side. Your JavaScript in turn must then return data as the output to the original YQL statement.

Ensuring the Security of Private Information


As mentioned earlier, a important feature of Open Data Tables is the ability to accommodate third-party security and authentication systems. As such, it is critical for developers to ensure an HTTPS connection is required in any case where "secret" or "private" information is being provided. If your table requires input that is deemed "private", such as any passwords, authentication keys, or other "secrets", you MUST ensure the https attribute within the table element is set to true. When YQL detects the https attribute is set to true, the table will no longer be usable for connections to the YQL console1 or to the Web service API. To test and use tables securely, you should now use the HTTPS endpoints: Console: https://developer.yahoo.com/yql/console Web Service API: https://query.yahooapis.com

Note
Connections made from the Open Data Table to the underlying Web services do not need to be made over HTTPS. The same holds true for the actual server hosting the Open Data Table definition. For more information on the https attribute within Open Data Tables, refer to "tables element" section within "Using Open Data Tables [2]".

JavaScript Objects, Methods, and Variables Reference


As you add JavaScript within your execute sub-element, you can take advantage of the following global objects: Object y [21] request [29] Description Global object that provides access to additional language capabilities. The request object is a global reference to the rest object [30] that contains the URL on which YQL would normally perform a GET request. You cannot reference the rest object directly, so you need to use request or y.rest to access the rest object. Response object returned as part of the "results" section of the YQL response. Global object that provides access to authorization credentials and cookies.

response [37] internal [38]

Let's discuss each of the global objects in detail.

http://developer.yahoo.com/yql/console

Yahoo! Developer Network

20

September 8, 2011

Executing JavaScript in Open Data Tables

y Global Object
The y global object contains methods that provide basic YQL functionality within JavaScript. It also allows you to include YQL Open Data Tables and JavaScript from remote sources. Method context Description Provides information about the context or environment that Javascript code is running in. Currently, one property is supported: context.table Returns Returns the Open Data Table name where JavaScript within an execute element has run.

crypto [22]

Provides basic cryptographic functions for Returns an encrypted string. use within JavaScript. Example: var md5string = y.crypto.encodeMd5("encode this string to md5");

date [22]

Provides access to date-related functions.

JavaScript object

Decodes a base64 string and then decom- Returns a string. d e c o m p r e s s ( b a s e 6 4 _ c o m - presses that string with gunzip. This method pressed_string) [24] allows you to decompress the POST request body and query parameters. diagnostics Returns diagnostic information related to the Returns diagnostic informacurrently executed script. tion.

e n v ( e n v i r o n m e n t Includes an environment file for use with your JavaScript. Example: file) y.env("http://datatables.org/alltables.env"); exit() include(url) jsonToXml(object) log(message) Stops the execution of the current script. Includes JavaScript located at a remote URL. Returns an evaluation of that include. Converts a JavaScript/JSON object into E4X object E4X/XML. Creates a log entry in diagnostics. Returns the log entry within the diagnostics output associated with the current select statement

Returns a JavaScript object when given a JavaScript object j s o n Parse(json_str) [25] well-formed JSON string. q u e r y ( s t a t e - Runs a YQL statement. ment) [27] q u e r y ( s t a t e m e n t , p a r a m s , timeout, callback) [27] rest(url, back) [29] A response object [37]that contains a result instance or an error object

Prepares and runs a YQL statement. Execute Creates a result instance, will replace all @name fields in the YQL or returns an error. statement with the values corresponding to the name in the supplied hashtable.

call- Sends a GET request to a remote URL end- point.

Yahoo! Developer Network

21

September 8, 2011

Executing JavaScript in Open Data Tables Method tidy (string html) use(url,namespace) xpath(object,xpath) xmlToJson(object) Description Tidy and return provided HTML. Returns Returns HTML that is run through HTML Tidy.

Imports an external Open Data Table defini- tion into the current script at runtime. Applies XPath to an E4X object. Returns a new E4X object Converts an E4X/XML object into a JSON JavaScript object object.

y.crypto functions
YQL provides several cryptographic functions for use within JavaScript. These functions reduce the need for external libraries and make YQL easier to use. Example: var md5string = y.crypto.encodeMd5("encode this string to md5"); Function Description Returns

encodeHmacSHA256(String Encrypts a string using HMAC- Returns an encrypted string. secret, String plaintext) SHA256 encryption. encodeHmacSHA1(String Encrypts a string using HMAC-SHA1 Returns an encrypted string. secret, String plaintext) encryption. encodeMd5(String text) encodeSha(String text) plain- Provides the MD5 hash of a string. Returns an MD5 hash.

plain- Provides the SHA-1 hash of a string. Returns an SHA-1 hash.

e n c o d e B a s e 6 4 ( S t r i n g Performs Base64 encoding of a string. Returns an Base64 encoded string. plaintext) d e c o d e B a s e 6 4 ( S t r i n g Performs Base64 decoding of a string. Returns an Base64 decoded string. plaintext) uuid() Provides a cryptographically secure Returns a UUID. version 4 Universal Unique Identifier (UUID).

y.date
Using y.date gives you access to date-related functions.

Methods
y.date.getOffsetFromEpochInMillis(time_format) Parses the given date string in a format specified by RFC 33392 and returns the number of milliseconds since the Unix epoch (0:00:00 UTC on January 1, 1970) as a string. Parameters:

http://www.ietf.org/rfc/rfc3339.txt

Yahoo! Developer Network

22

September 8, 2011

Executing JavaScript in Open Data Tables time_format <String> A string in the form of the time/date formats or regex patterns below. The regex patterns are case insensitive. RFC 3339 Timestamp3 ^([+-])?\s*(\d+)\s*(year|month|week|day|hour|minute)s?$ ^(\d+)\s*(year|month|week|day|hour|minute)s?\s+(from\s+now|after|after\s+today|later|old|ago)$ ^now\s+([+-])\s*(\d+)\\s+(year|month|week|day|hour|minute)s?$ ^last\s+(year|month|week)$ now yesterday tomorrow Returns: String Examples: In the example table below, getOffsetFromEpochInMills is used in conjunction with the JavaScript Date object to log several different times. The table then returns the execution time of the JavaScript in the response. ... <execute><![CDATA[ var start_time = parseInt(y.date.getOffsetFromEpochInMillis("now")); var start_date = new Date(start_time); var yesterday_time = parseInt(y.date.getOffsetFromEpochInMillis("yesterday")); var yesterday_date = new Date(yesterday_time); var month_from_now = new Date(parseInt(y.date.getOffsetFromEpochInMillis("now + 1 month"))); var ali_liston_fight = new Date(parseInt(y.date.getOffsetFromEpochInMillis("1964-02-25"))); y.log("Start Time: " + start_date.toDateString()); y.log("Yesterday: " + yesterday_date.toDateString()); y.log("Last Month: " + month_from_now.toDateString()); y.log("Ali beats Liston: " + ali_liston_fight.toDateString()); response.object = "Execution Time: " + String((parseInt(y.date.getOffsetFromEpochInMillis("now")) parseInt(start_time))/1000) + " seconds."; ]]> </execute> ... In this example XML response returned by the above table, the diagnostic information includes the logged times, and the result element contains the execution time.

http://www.ietf.org/rfc/rfc3339.txt

Yahoo! Developer Network

23

September 8, 2011

Executing JavaScript in Open Data Tables <query xmlns:yahoo="http://www.yahooapis.com/v1/base.rng" yahoo:count="1" yahoo:created="2011-07-12T02:20:59Z" yahoo:lang="en-US"> <diagnostics> <publiclyCallable>true</publiclyCallable> <url execution-time="76"> <![CDATA[http://zhouyaoji.com/yql/date.xml]]> </url> <log>Start Time: Tue Jul 12 2011</log> <log>Yesterday: Mon Jul 11 2011</log> <log>Last Month: Fri Aug 12 2011</log> <log>Ali beats Liston: Tue Feb 25 1964</log> <javascript execution-time="6" instructions-used="12134" table-name="sc"/> <user-time>89</user-time> <service-time>76</service-time> <build-version>19521</build-version> </diagnostics> <results> <result>Execution Time: 0.004 seconds.</result> </results> </query>

y.decompress(base64_compressed_string)
Allows you to decompress query parameters and the HTTP POST request body. The decompress method decodes a base64 string and then uses gunzip to decompress the string. Parameters: base64_compressed_string <String> A string that has been compressed with gzip and then base64 encoded. Returns: String Examples: YQL Statement: use 'http://example.com/decompress.xml' as decompress; select * from decompress where foo="H4sIAOQTlU0AA8tIzcnJBwCGphA2BQAAAA==" Example YQL Open Data Table: http://example.com/decompress.xml The table decompress.xml below uses y.decompress to decompress the value assigned to foo in the YQL statement above and then returns it as the string. <?xml version="1.0" encoding="UTF-8"?> <table xmlns="http://query.yahooapis.com/v1/schema/table.xsd" securityLevel="any"> <bindings> <select itemPath="" produces="XML"> <inputs> <key id="foo" type="xs:string" paramType="variable"required="true"/> </inputs> <execute><![CDATA[

Yahoo! Developer Network

24

September 8, 2011

Executing JavaScript in Open Data Tables try { y.log(foo); var bar = y.decompress(foo); } catch (err) { y.log(err.message); } response.object = bar; ]]></execute> </select> </bindings> </table> Returned XML Response: <?xml version="1.0" encoding="UTF-8"?> <query xmlns:yahoo="http://www.yahooapis.com/v1/base.rng" yahoo:count="1"yahoo:created="2011-04-01T00:06:38Z" yahoo:lang="en-US"> <diagnostics> <publiclyCallable>true</publiclyCallable> <url execution-time="2"><![CDATA[http://example.com/decompress.xml]]></url> <log>H4sIAOQTlU0AA8tIzcnJBwCGphA2BQAAAA==</log> <javascript execution-time="11" instructions-used="0"table-name="d"/> <user-time>42</user-time> <service-time>2</service-time> <build-version>12012</build-version> </diagnostics> <results> <result>hello</result> </results> </query>

y.jsonParse(json_str)
Parses a well-formed JSON string and returns the resulting JavaScript object. If given a malformed JSON string, y.jsonParse will throw a runtime JSONException. Parameters: json_str <String> A well-formed JSON string. Returns: Object Examples: YQL Statement: use 'http://example.com/parseJson.xml' as jp; select * from jp where mykey="sample_key" Example YQL Open Data Table: http://example.com/parseJson.xml The table parseJson.xml below parses a JSON string and returns a JavaScript object. <table ...

Yahoo! Developer Network

25

September 8, 2011

Executing JavaScript in Open Data Tables <bindings> <select itemPath="" produces="XML"> <inputs> <key id="mykey" type="xs:string" paramType="variable" required="true"/> </inputs> <execute><![CDATA[ var mycontent ='\{ "firstName": "John",\ "lastName": "Smith",\ "address": {\ "streetAddress": "21 2nd Street",\ "city": "New York",\ "state": "NY",\ "postalCode": "10021"\ },\ "phoneNumbers": [\ {\ "type": "home",\ "number": "212 555-1234"\ },\ {\ "type": "fax",\ "number": "646 555-4567"\ }\ ]\ }'; y.log(mycontent); j= y.parseJson(mycontent); y.log(j); response.object=j.get('lastName'); ]]></execute> </select> </bindings> </table> Example XML Response: <query xmlns:yahoo="http://www.yahooapis.com/v1/base.rng" yahoo:count="0" yahoo:created="2011-06-15T23:23:21Z" yahoo:lang="en-US"> <diagnostics> <publiclyCallable>true</publiclyCallable> <url execution-time="2" proxy="DEFAULT"><![CDATA[http://example.com/parseJson.xml]]></url> <log>{ "firstName": "John", "lastName": "Smith", "address": { "streetAddress": "21 2nd Street", "city": "New York", "state": "NY", "postalCode": "10021" },

Yahoo! Developer Network

26

September 8, 2011

Executing JavaScript in Open Data Tables "phoneNumbers": [ { "type": "home", "number": "212 555-1234" }, { "type": "fax", "number": "646 555-4567" } ] }</log> <log>{ "firstName":"John", "lastName":"Smith", "address":{ "streetAddress":"21 2nd Street", "city":"New York", "state":"NY", "postalCode":"10021" }, "phoneNumbers": [ { "type":"home", "number":"212 555-1234" }, { "type":"fax", "number":"646 555-4567" } ] }</log> <javascript execution-time="2" instructions-used="0" table-name="xxx"/> <user-time>8</user-time> <service-time>2</service-time> <build-version>18749</build-version> </diagnostics> <results> <result>Smith</result> </results> </query>

y.query(statement, params, timeout, callback) method


Perhaps you want to use YQL queries while still using JavaScript within YQL. y.query allows you to perform additional YQL queries within the execute sub-element. Parameters: statement <String> A YQL statement. params <Object> The object can contain key-values that can be referenced in the YQL statement for variable substitution. In the example below, the @url variable in the passed YQL statement is replaced with the value associated with the url key of the params object.

Yahoo! Developer Network

27

September 8, 2011

Executing JavaScript in Open Data Tables var q = y.query('select * from html where url=@url and xpath="//div[@id=\'yfi_headlines\']/div[2]/ul/li/a"', {url:'http://finance.yahoo.com/q?s=yhoo'}); var results = q.results; timeout <Number> The number of milliseconds before the call times out. callback <Function> The callback function to handle the returned response. See Making Asynchronous Calls with JavaScript Execute [47] for more information about using the callback. Returns: Object The returned object contains the following properties: Property results query timeout The results. The YQL statement that was passed as a parameter to y.query. diagnostics The diagnostics. Description Data Type E4X object E4X object string

Specifies the request timeout in milliseconds. This is useful when you None want to cancel requests that take longer than expected.

Example: In the following code snippet, y.query executes a statement that uses the html table to get Yahoo! financial data. var q = y.query('select * from html where url="http://finance.yahoo.com/q?s=yhoo" and xpath="//div[@id=\'yfi_headlines\']/div[2]/ul/li/a"'); var results = q.results; Below the y.query method is passed a callback function that allows asynchronous calls. // callback function logs the order in // which url requests completed. var mycallback = function(result) { count++; y.log("results received for query: " +result.query); if (first == null) { first = result.response; } } // Create 4 asynchronous y.rest requests. y.query("select * from flickr.photos.search where has_geo='true' and text='san francisco'", mycallback); y.query("select * from feednormalizer where url='http://rss.news.yahoo.com/rss/topstories' and output='atom_1.0'", mycallback); y.query("select * from weather.forecast where location=90210", mycallback); y.query("select * from answers.search where query="cars" and category_id=2115500137 and type="resolved"", mycallback);

Yahoo! Developer Network

28

September 8, 2011

Executing JavaScript in Open Data Tables Queries called from y.query return and execute instantly. However, data is only returned when the results property is accessed. This feature allows you to make multiple, independent queries simultaneously that are then allowed to process before being returned together when the results property is accessed.

y.rest(url, callback) method


The y.rest method allows you to make GET requests to remote Web services. It also allows you to pass parameters and headers in your request and use a callback function to make asynchronous requests. See Making Asynchronous Calls with JavaScript Execute [47] for more information about using the callback. Parameters: url <String> The URL endpoint to a query. callback <Function> (Optional) The callback mechanism waits to receive returned results that it can then process. This callback function allows asynchronous calls because the function can wait for the returned results while y.rest is called again. Returns: rest Object [30] Examples: In the following code snippet, an HTTP GET request is made to example.com and the response is saved to data. var myRequest = y.rest('http://example.com'); var data = myRequest.get().response; By passing a callback to y.rest to handle the returned response as seen here, you can make asynchronous calls. // Callback function logs the order in which URL // requests completed. var mycallback = function(result) { count++; y.log("response received for url: " +result.url + ", Status: " + result.status + ", total response count: " + count); if (first == null) { first = result.response; } } // Create 4 asynchronous y.rest requests. y.rest("http://www.yahoo.com", mycallback).get(); y.rest("http://finance.yahoo.com", mycallback).get(); y.rest("http://movies.yahoo.com", mycallback).get(); y.rest("http://news.yahoo.com", mycallback).get();

Tip
The y.rest method supports "chaining", which means that you can construct and run an entire REST request by creating a "chain" of methods. Here is a hypothetical example:

Yahoo! Developer Network

29

September 8, 2011

Executing JavaScript in Open Data Tables var myData = y.rest('http://blah.com') .path("one") .path("two").query("a","b") .header("X-TEST","value") .get().response; When chained, the resulting request looks like this: http://blah.com/one/two?a=b

Note
Because JSON does not have a "root" node in most cases, all JSON responses from a remote Web service will be contained within a special json root object under response.results.

request Global Object


The request global object is essentially a reference to a rest object [30]. You must use the request object, y.rest, or y.query to access the rest object.

rest Object
The rest object has properties and methods for getting information and making REST calls.

Properties
Property headers url queryParams Description Gets the hashmap of headers Provides a URL endpoint to query object string Data Type

Gets the hashmap of query parameters E4X object containing query parameters.

matrixParams Gets the hashmap of matrix parameters result object [37] containing matrix parameters.

Methods
accept(content-type)
Specifies the type of content to send in the response using the Accept HTTP header. This tells YQL what kind of data format you want returned, as well as how to parse it. Parameters: content-type <String> The content type of the data being sent. For example: application/xml. Returns: rest Object [29] Examples:

Yahoo! Developer Network

30

September 8, 2011

Executing JavaScript in Open Data Tables The following is an example of how you would specify JSON as the return format for data you send as XML: var ret = request.accept('application/json').contentType('application/xml').post(content).response; Using the above example, YQL will convert XML to JSON prior to returning it in the response.

contentType(content-type)
Specifies the content-type of the data being sent. An example of a content-type is: application/json. This object is useful with INSERT and UPDATE statements.

Note
This method does not automatically convert one data format to another prior to sending it, so if you indicate one format in contentType but actually send another, your Web service may produce an error. Parameters: content-type <String> The content-type of the data being sent. An example of a content-type is: application/json. Returns: rest Object The following is an example of how you would specify XML as the data you are sending: var content = <employee> <name>John</name> <id>21</id> </employee>; var ret = request.contentType('application/xml').post(content).response;

decompress(bool)
Allows you to configure a REST call to decompress the returned response with gunzip. Parameters: boolean <Boolean> Returns: rest Object [29] Examples: In this code example, the header method sets the HTTP header Accept-Encoding to gzip, and then the decompress method is used to decompress the returned response. var resp = y.rest("www.example.com").header("Accept-Encoding","gzip").decompress(true).get(); response.object = resp;

Yahoo! Developer Network

31

September 8, 2011

Executing JavaScript in Open Data Tables

del()
Performs an HTTP DELETE. This object is useful with DELETE statements. Parameters: None Returns: result Object [37] Examples: In the below Open Data Table, the del() method is used to delete the Mixi voice status identified by the post ID. Using del() sends an HTTP DELETE request to the URL defined in the url element. <delete itemPath="" produces="JSON"> <urls> <url>http://api.mixi-platform.com/2/voice/statuses/{post_id}</url> </urls> <inputs> <key id="oauth_token" type="xs:string" paramType="query" required="true" /> <key id="post_id" type="xs:string" paramType="path" required="true" /> </inputs><execute><![CDATA[response.object = request.del().response;]]></execute> </delete>

fallbackCharset(charset_list)
Overrides the list of fallback character sets, which is set to "utf-8, iso-8859-1" by default, for decoding the returned response. YQL attempts to decode the response using the character sets listed in charset_list when the response either does not specify the character set or specifes an incorrect character set that results in a failed decoding. Parameters: charset_list <String> A list of one or more fallback character sets. For example: "ISO-88591, utf-8" Response: result Object [37] Examples: response.object = y.rest(url).fallbackCharset('shift_jis,iso-8859-8, UTF-8, iso-8859-1').get().response;

forceCharset(charset)
Forces YQL to use the character set specified by charset. Using this method overrides both the character set specified by the response and the fallback character sets. Parameters:

Yahoo! Developer Network

32

September 8, 2011

Executing JavaScript in Open Data Tables charset <String> The character encoding4, such as "ISO-8559-1", that YQL will use to render a response. Returns: rest Object [29] Examples: y.rest('www.uol.com.br').get().response.forceCharset('iso-8859-1');

get()
Performs a GET request to the URL endpoint. This object is useful with SELECT statements. Parameters: None Returns: result Object [37] Examples: r = y.rest("yahoo.com").contentType('application/json').get(); response.object = r;

header(name, value)
Adds an HTTP header to the request. Parameters: name <String> The name of the HTTP header. value <String> The value associated with the HTTP header. Returns: rest Object [30] Examples: The header method in this code snippet is used to pass authorization credentials to get a response. y.include("http://yqlblog.net/samples/base64.js"); var authheader = "Basic "+Base64.encode(username+":"+password); response.object = request.header("Authorization",authheader).get().response; In this code snippet, the header method sets the HTTP header Content-Encoding to gzip, so that the returned response will be compressed. var resp = y.rest("http://www.apache.org").header("Accept-Encoding","gzip").get(); response.object = resp.response;

matrix(name,value)
Adds a matrix parameter to the request.
4

http://en.wikipedia.org/wiki/Character_encoding#Common_character_encodings

Yahoo! Developer Network

33

September 8, 2011

Executing JavaScript in Open Data Tables Parameters: name <String> value <String> Returns: rest Object [30] Examples: The code snippet below posts the matrix parameters to flower_order_server.com. y.rest("http://flower_order_service.com").matrix('flower','roses').matrix('color','red').post(); Matrix parameters are attached to the URI resource like query parameters, but are delimited by a semicolon and not an ampersand and the matrix parameters are not separted from the URI resource by a question mark. The yql.rest call creates the following URL: http://flower_order_service.com;flower=roses;color=red

path(path_segment)
Appends a path segment to the URI. Parameters: path_segment <String> The path segment to add to a URI. Returns: request Object [30] Examples: var myData = y.rest('http://blah.com') .path("one") .path("two") .query("a","b") .header("X-TEST","value") .get().response;

post(content)
Performs an HTTP POST, using the value of the content, to the URL endpoint. This object is useful with INSERT, UPDATE, and DELETE statements. Parameters: content <String|Object> The data that is posted. Returns: result Object [37] Examples: var content = { "employee": { "name": "John", "id": 21 }}; var ret = request.contentType('application/json').post(content).response;

proxy(proxy_string)
Sets a proxy from an internal data table. The proxy method allows you to do the following:

Yahoo! Developer Network

34

September 8, 2011

Executing JavaScript in Open Data Tables cache results in the YSquid5 cache layer hosted by YQL. anonymize Yahoo! data before sending traffic out of the network. use an extended cache so that a request is routed by the External Cache6 hosted by Yahoo! Edge Services. Parameters: proxy_string <String> Either a hostname denoting a standard HTTP proxy that can be hosted by the developing property and accessible from the YQL servers or one of the following proxy settings that specifies the proxy and cache to use: "default" - the Yahoo! HTTP Proxy7 "internal" - the YSquid cache layer hosted by YQL. "long30min" - the 30-minute extended cache hosted for YQL. "long1day" - the one-day extended cache. "long7day" - the seven-day extended cache. Examples: The example table below uses the default proxy (Yahoo! HTTP Proxy) to make a REST call to an external server. Using this proxy, Yahoo! data can be anonymized before being sent out of the network. <?xml version="1.0" encoding="UTF-8"?> <i:table xmlns:i="http://query.yahooapis.com/v1/schema/internalTable.xsd" xmlns="http://query.yahooapis.com/v1/schema/table.xsd" securityLevel="any"> <i:bindings> <i:select itemPath="" produces="XML"> <execute><![CDATA[ var url = 'http://www.external-server.com/somepage'; response.object = y.rest(url).proxy('default').get(); ]]></execute> </i:select> </i:bindings> </i:table> In this example table, the internal proxy setting allows the caching of results from Yahoo! Finance to the Squid Cache layer provided by the YQL team. <?xml version="1.0" encoding="UTF-8"?> <i:table xmlns:i="http://query.yahooapis.com/v1/schema/internalTable.xsd" xmlns="http://query.yahooapis.com/v1/schema/table.xsd" securityLevel="any"> <i:bindings> <i:select itemPath="" produces="XML"> <execute><![CDATA[ var url = 'http://finance.yahoo.com/q?s=yhoo';
5 6

http://twiki.corp.yahoo.com/view/Platform/YahooSquid http://twiki.corp.yahoo.com/view/Platform/YTSExternalCache 7 http://twiki.corp.yahoo.com/view/TrafficManagement/HtProxy

Yahoo! Developer Network

35

September 8, 2011

Executing JavaScript in Open Data Tables response.object = y.rest(url).proxy('internal').get(); ]]></execute> </i:select> </i:bindings> </i:table> Because the Yahoo! News RSS feed does not change quickly, the proxy is set to 'long30min', which will force the request to be routed using the External Cache hosted by Yahoo! Edge Services. <?xml version="1.0" encoding="UTF-8"?> <i:table xmlns:i="http://query.yahooapis.com/v1/schema/internalTable.xsd" xmlns="http://query.yahooapis.com/v1/schema/table.xsd" securityLevel="any"> <i:bindings> <i:select itemPath="" produces="XML"> <execute><![CDATA[ var url = 'http://news.yahoo.com/rss/us'; response.object = y.rest(url).proxy('long30min').get(); ]]></execute> </i:select> </i:bindings> </i:table>

put(content)
Performs an HTTP PUT, using the value of the content, to the URL endpoint. This object is useful with INSERT, UPDATE, and DELETE statements. Parameters: content <String | Object> The data that is used in a PUT request. Returns: result Object [37] Examples: url = blogurl + "/xmlrpc.php"; myRequest = y.rest(url); myRequest.contentType("text/xml"); results = myRequest.put('<?xml version="1.0" encoding="UTF-8"?>' + postData.toString()).response;

query(key, value)
Adds a single query parameter. Parameters: key <String> value <String> Returns: request Object [30] Examples:

Yahoo! Developer Network

36

September 8, 2011

Executing JavaScript in Open Data Tables var formUrl = "http://example.com/form"; var resp = y.rest(formUrl).query("name","Tom").post().response;

query(hashmap)
Adds all the query parameters based on key-name hashmap. Parameters: hashmap <Object> Returns: request Object [30] Examples: var formUrl = "http://example.com/form"; var name = { "name": "John" }; var resp = y.rest(formUrl).query(name).post().response;

timeout(milli_seconds)
Specifies the request timeout in milliseconds. This is useful when you want to cancel requests that take longer than expected. Parameters: milli_seconds <Number> The number of seconds before your request times out. Returns: None Examples: // timeout after 500 milliseconds y.rest('http://....').timeout(500).get();

response Global Object


The response global object allows you to determine how responses are handled and is also used to set the value to the data you'd like to return, as an E4X object, a JSON structure, or simply a string. Object Description

object Contains the results of your execute script. Set this value to the data you'd like to return, as an E4X object, a JSON structure, or simply a string.

result Object
The result object contains the results of your execute script. The table below lists the properties of the result object. Property response [37] Description Data Type

Get the response from the remote service. If the response con- E4X object or string tent type is not application/json or text/xml then

Yahoo! Developer Network

37

September 8, 2011

Executing JavaScript in Open Data Tables Property Description YQL provides a string. If JSON or XML is specified, the E4X representation of the data is returned. headers status timeout url The headers returned from the response. The HTTP status code. Indicates if the call timed out. The URL used to make the request. object string boolean string Data Type

internal Global Object


The internal global object is only available for you by Yahoo! engineers. From the internal object, you can create B-Cookies, set cookies, get cookies, and access authorization credentials. See Accessing Cookies from JavaScript Execute [27] in the YQL for Yahoos for more information. The table below lists the properties of the internal object. Property auth headers request Description Contains a list of internal header information and authorization credentials. The internal headers returned from the response. Data Type array object

Internal object that provides access to information about the request, such as object the cookies that were sent.

response Internal object that provides access to information about the response headers object and body. yck The namespace that gives you access to some functionality of the Yahoo! object Cookie Library (YCK).

Methods
setCookie(name, value,expires)
Sets a cookie for the specified duration. Parameters: name <String> The name of the cookie. value <String> The value of the cookie. expires <Number> Time in seconds that the cookie will expire. Returns: Null Examples: The following is an example of how you would set the cookie visited to true for 10 minutes: internal.setCookie("visited", "true",600); Because the return value is null, you should use a try-catch block to set cookies to catch exceptions as seen below:

Yahoo! Developer Network

38

September 8, 2011

Executing JavaScript in Open Data Tables try { internal.setCookie('site', 'www.yahoo.com',1000); { y.log('exception caught'); y.log(err); }

} catch(err)

auth Array
The auth array contains objects and key-value pairs containing information about the internal headers and authorization credentials. The following table lists some of the most important properties contained in the auth array. Property guid lang Description The GUID of the user making the request to YQL. Data Type array string

cookie All of the cookie data passed in the HTTP headers.

The human language of the returned response. Examples: de-DE, de-AT, de-BE. string The value of this attribute is the same as that of the Content-Language header. The value is an RFC 46468 language tag.

Examples
The following example sets a cookie with the GUID and language of the response and then returns all of the cookies to response.object. <?xml version="1.0"?> <i:table xmlns:i="http://query.yahooapis.com/v1/schema/internalTable.xsd" xmlns="http://query.yahooapis.com/v1/schema/table.xsd"> <i:bindings> <i:select produces="JSON" itemPath=""> <execute><![CDATA[ var guid = internal.auth.guid; var lang = internal.auth.lang; internal.setCookie("guid",guid,300); internal.setCookie("lang",lang,300); response.object = internal.auth.cookie; ]]> </execute> </i:select> </i:bindings> </i:table>

request Object
The internal request object allows access to information about the request and to get cookies. Property Description Data Type

cookies All of the cookie data passed in the HTTP headers. object

http://www.ietf.org/rfc/rfc4646.txt

Yahoo! Developer Network

39

September 8, 2011

Executing JavaScript in Open Data Tables

Examples
The example table below stores the cookies returned by internal.request.cookies and assigns it to response.object. <?xml version="1.0"?> <i:table xmlns:i="http://query.yahooapis.com/v1/schema/internalTable.xsd" xmlns="http://query.yahooapis.com/v1/schema/table.xsd"> <i:bindings> <i:select produces="JSON" itemPath=""> <execute><![CDATA[ var cookies = internal.request.cookies; var strCookie = ''; for(var i=0; i<cookies.length; i++) { strCookie += cookies[i].name + '=' + cookies[i].value + ';'; if(i<cookies.length-1) strCookie += ' '; } response.object = strCookie; ]]> </execute> </i:select> </i:bindings> </i:table>

response Object
Property Description Data Type cookies All of the cookie data passed in the HTTP headers. object

Methods
containsHeader(header_name) Determines whether the specified HTTP header was found. Parameters: header_name <String> The name of the HTTP header. For example: "Set-Cookie" Returns: Boolean Examples: The following example table sets cookies and then confirms that the HTTP header contains "SetCookie". <i:table xmlns:i="http://query.yahooapis.com/v1/schema/internalTable.xsd" xmlns="http://query.yahooapis.com/v1/schema/table.xsd"> <i:bindings> <i:select produces="JSON" itemPath=""> <execute><![CDATA[ internal.setCookie("yql",true,500); if(internal.response.containsHeader("Set-Cookie")){ response.object = "Cookies were set.";

Yahoo! Developer Network

40

September 8, 2011

Executing JavaScript in Open Data Tables }else { response.object = "Cookies weren't set."; }]]> </execute> </i:select> </i:bindings> </i:table> setHeader(header_name, header_value) Allows you to set an HTTP header. Parameters: header_name <String> The name of the HTTP header. header_value <String> The value given to the HTTP header specified by header_name. Returns: Null Examples: The example table below sets the HTTP header YQL and then confirms that the header was set. <i:table xmlns:i="http://query.yahooapis.com/v1/schema/internalTable.xsd" xmlns="http://query.yahooapis.com/v1/schema/table.xsd"> <i:bindings> <i:select produces="JSON" itemPath=""> <execute><![CDATA[ internal.response.setHeader("YQL", true); if(internal.response.containsHeader("YQL")) { response.object = "Response created by YQL."; } ]]></execute> </i:select> </i:bindings> </i:table>

yck Object
The yck object allows you to create Yahoo! B-Cookies9 that are used for browser identification and ad targeting. See the twiki Yahoo Cookies :: B-Cookie10 for more information.

Methods
createBCookie() Creates the Yahoo! B-Cookie. Parameters: None

10

http://twiki.corp.yahoo.com/view/Platform/YahooCookieB http://twiki.corp.yahoo.com/view/Platform/YahooCookieB

Yahoo! Developer Network

41

September 8, 2011

Executing JavaScript in Open Data Tables Returns: Object Examples: The following table creates the B-Cookie and then passes it via the HTTP headers. <?xml version="1.0"?> <i:table xmlns:i="http://query.yahooapis.com/v1/schema/internalTable.xsd" xmlns="http://query.yahooapis.com/v1/schema/table.xsd"> <i:bindings> <i:select produces="JSON" itemPath=""> <execute><![CDATA[ var b = internal.yck.createBCookie(); internal.response.setHeaders("Set-Cookie", b.toString());]]> </execute> </i:select> </i:bindings> </i:table>

Global Variables
The following global variables are available for use within the execute element of YQL Open Data Tables: Variable input Description This global variable is available for each binding within the inputs element such as key, value, or map. For example, to call the first binding below you would use nameofid. var mycontent = nameofid;

Important
If the id name uses an illegal identifier, such as the use of hyphens, you must instead use the inputs global variable. <inputs> <key id="nameofid" type="xs:string" paramType="path" default="my default key" required="true" /> <value id="field-name" type="xs:string" paramType="variable" default="my default field" /> </inputs> When a map binding is present in the Open Data Table, the global variable is present as a named hashtable. Each value provided in the YQL statement is set in the hashmap. inputs This global variable is an array that contains each binding within the inputs element, along with its value. For example, to call the second binding above, you would use inputs[fieldname]: var mycontent =inputs['field-name'];

Yahoo! Developer Network

42

September 8, 2011

Executing JavaScript in Open Data Tables

JavaScript and E4X Best Practices for YQL


The following is a series of best practices related to using JavaScript and E4X within the execute subelement in YQL: Paging Results [43] Including Useful JavaScript Libraries [43] Using E4X within YQL [44] Logging and Debugging [46]

Paging Results
YQL handles paging of returned data differently depending on how you control paging within an Open Data Table definition. Let us consider the following example, followed be three paging element scenarios: select * from table(10,100) where local.filter>4.0 No page element: If no paging element is provided, YQL assumes you want all data available to be returned at once. Any "remote" paging information provided on the select (10 being the offset and 100 being the count in our example), will be applied to all of the results before being processed by the remainder of the where clause. In our example above, the first 10 items will be discarded and only another 100 will be used, and execute will only be called once. A paging element that only supports a variable number of results: If a paging element is provided that only supports a variable number of results (a single page with variable count), then the execute sub-element will only be called once, with the total number of elements needed in the variable representing the count. The offset will always be 0. In our example, the count will be 110, and the offset 0. A paging element that supports both offset and count: If a paging element is provided that supports both offset and count, then the execute sub-element will be called for each "page" until it returns fewer results than the paging size. In this case, let's assume the paging size is 10. The execute subelement will be called up to 10 times, and expected to return 10 items each time. If fewer results are returned, paging will stop.

Note
In most cases, paging within the Open Data Table should match the paging capabilities of the underlying data source that the table is using. However, if the execute sub-element is adjusting the number of results coming back from a fully paging Web service or source, then there is usually no way to unify the "offset" of the page as set up in the Open Data Table with the destinations "offset". You may need to declare your Open Data Table as only supporting a variable number of results in this situation.

Including Useful JavaScript Libraries


When writing your execute code, you may find the following JavaScript libraries useful: OAuth:

Yahoo! Developer Network

43

September 8, 2011

Executing JavaScript in Open Data Tables y.include("http://oauth.googlecode.com/svn/code/javascript/oauth.js"); y.include("http://oauth.googlecode.com/svn/code/javascript/sha1.js"); Flickr: y.include("http://blog.pipes.yahoo.net/wp-content/uploads/flickr.js"); MD5, SHA1, Base64, and other Utility Functions11: y.include("http://v8cgi.googlecode.com/svn/trunk/lib/util.js");

Using E4X within YQL


ECMAScript for XML (simply referred to as E4X) is a standard extension to JavaScript that provides native XML support. Here are some benefits to using E4X versus other formats, such as JSON: Preserves all of the information in an XML document, such as namespaces and interleaved text elements. Since most web services return XML this is optimal. You can use E4X selectors and filters to find and extract parts of XML structure. The engine on which YQL is created natively supports E4X, allowing E4X-based data manipulation to be faster. Supports XML literals, namespaces, and qualified names. To learn more about E4X, refer to these sources online: E4X Quickstart Guide12 from WS02 Oxygen Tank Processing XML with E4X13 from Mozilla AJAX and scripting Web service with E4X14 by IBM Introducing E4X15 by O'Reilly Popular E4X Bookmarks16 by delicious E4X guide17 by rephrase.net

E4X Techniques
In addition to the resources above, the following tables provides a quick list of tips related to using E4X:

11 12

http://code.google.com/p/v8cgi/wiki/API_Util http://wso2.org/project/mashup/0.2/docs/e4xquickstart.html 13 https://developer.mozilla.org/en/Core_JavaScript_1.5_Guide/Processing_XML_with_E4X 14 http://www.ibm.com/developerworks/webservices/library/ws-ajax1/ 15 http://www.xml.com/pub/a/2007/11/28/introducing-e4x.html 16 http://delicious.com/popular/e4x 17 http://rephrase.net/days/07/06/e4x

Yahoo! Developer Network

44

September 8, 2011

Executing JavaScript in Open Data Tables E4X Technique Creating XML literals Substituting variables Notes Code Example v a r x m l <root>hello</root>; var y = =

Use curly brackets {} to substi- var x = "text"; tute variables. You can use this <item>{x}</item>; for E4X XML literals as well.

Adding sub-elements to an When adding sub-elements to an item.node+=<subel></subel>; element element, include the root node for the element. You can add a sub-element to a x.node += <sub></sub>; node in a manner similar to adding sub-elements to an ele- This above code results in the following structure: ment. <node><sub></sub></node> If you try to add a sub-element to x += <sub></sub>; a node without including the root node, you will simply append the The above code results in the following element and create an XML list. structure: <node><node><sub></sub>; Assigning variably named Use substitution in order to create var item = <{name}/>; elements an element from a variable. Assigning a value to an at- tribute Getting all the elements within a given element Getting specific objects within an object anywhere under a node Getting the immediate H3 children of an element Getting an attribute of an element item.@["id"]=path[0]; var hs2 = el..*; var hs2 = el..div

h2 = el.h3; h3 = el.h3.@id; or h3 = el.h3.@["id"];

Getting elements with a certain attribute

var alltn15divs = d..div.(@['id'] =="tn15content");

Getting the "class" attrib- Use brackets to surround the className =t.@['class']; ute "class" attribute. Getting a class as a string To get a class as a string, get its var classString = classtext object and the apply to- Name.text().toString() String.

Getting the name of a node Use localName() to get the var nodeName = e4xnode.locname of a node. alName();

Yahoo! Developer Network

45

September 8, 2011

Executing JavaScript in Open Data Tables

Note
When using E4X, note that you can use XML literals to insert XML "in-line," which means, among other things, you do not need to use quotation marks: var myXml = <foo />;

E4X and Namespaces


When working with E4X, you should know that E4X objects are namespace aware. This means that you must specify the namespace before you work with E4X objects within that namespace. The folllowing example sets the default namespace: default xml namespace ='http://www.inktomi.com/'; After you specify a default namespace, all new XML objects will inherit that namespace unless you specify another namespace.

Caution
If you do not specify the namespace, elements will seem to be unavailable within the object as they reside in a different namespace.

Tip
To clear a namespace, simply specify a blank namespace: default xml namespace ='';

JavaScript Logging and Debugging


To get a better understanding of how your executions are behaving, you can log diagnostic and debugging information using the y.log statement along with the y.getDiagnostics element to keep track of things such as syntax errors or uncaught exceptions. The following example logs "hello" along with a variable: y.log("hello"); y.log(somevariable); Using y.log allows you to get a "dump" of data as it stands so that you can ensure, for example, that the right URLs are being created or responses returned. The output of y.log goes into the YQL diagnostics element when the table is used in a select. You can also use the follow JavaScript to get the diagostics that have been created so far: var e4xObject = y.getDiagnostics();

Executing JavaScript Globally


When the execute element is placed outside of the binding element, it is available globally within an Open Data Table and is usable across each of the table's bindings. For example, in the following Open

Yahoo! Developer Network

46

September 8, 2011

Executing JavaScript in Open Data Tables Data Table, the function arrayConcat is available to be used within the execute element for SELECT statements: <?xml version="1.0" encoding="UTF-8"?> <table xmlns="http://query.yahooapis.com/v1/schema/table.xsd" securityLevel="any"> <meta> <author>liang</author> <description>execute block outside bindings</description> <documentationURL/> </meta> <execute><![CDATA[ function arrayConcat(parents, children) { var family=parents.concat(children); return family; } ]]></execute> <bindings> <select itemPath="" produces="XML"> <urls> <url env="all">http://fake.com</url> </urls> <inputs> <key id="mykey" type="xs:string" paramType="variable" required="true" /> </inputs> <execute><![CDATA[ myparents=["Sister", "Brother"]; mychildren=["Dad", "Mom"]; var myfamily = arrayConcat(myparents,mychildren); response.object = <family>{myfamily}</family>; ]]></execute> </select> </bindings> </table>

Making Asynchronous Calls with JavaScript Execute


The y.rest and y.query functions take callback functions as a parameter. The callback function is called either when y.rest or y.query completes a call or times out. This callback mechanism allows for simultaneously handling and processing multiple REST calls. The callback functions are just JavaScript functions that are passed the results of the REST call. When the callback function below is passed as a parameter to y.rest, the response is assigned to result.

Yahoo! Developer Network

47

September 8, 2011

Executing JavaScript in Open Data Tables The code example shows the basic syntax of using y.rest with a callback function. You can see how the callback can be implemented in Examples. function callback(result) { // do something with 'result' } y.rest("http://example.com",callback);

Examples
y.rest
The returned result of y.rest has the two properties timeout and url that allow you to identify calls that have timed out and retain the URI resource. The two properties can also be accessed by the callback function. A single callback function can be used for multiple requests. The following example uses the same callback to log the URL and status of the response as well as the number of responses received. // callback function logs the order in which // url requests completed. var mycallback = function(result) { count++; y.log("response received for url: " +result.url + ", Status: " + result.status + ", total response count: " + count); if (first == null) { first = result.response; } } // create 4 asynchronous 'y.rest' requests. y.rest("http://www.yahoo.com", mycallback).get(); y.rest("http://finance.yahoo.com", mycallback).get(); y.rest("http://movies.yahoo.com", mycallback).get(); y.rest("http://news.yahoo.com", mycallback).get(); The two properties url and timeout of the response help you determine if a call has timed out or associate the response with the original URL. The code snippet below show you the url and timeout properties and how you could potentially use them: // The 'url' property allows you to connect // the original request url to the response var uri = y.rest("http://www.yahoo.com").get().url // The 'timeout' property allows you to make another call // or take other action if your original call times out var timed_out = y.rest('http://www.yahoo.com').get().timeout if(timed_out) { // Try one more time var resp = y.rest(uri).get().response; } The callback function can also use the url and timeout properties to take the appropriate action as seen in this code snippet:

Yahoo! Developer Network

48

September 8, 2011

Executing JavaScript in Open Data Tables // This callback takes advantage of the // 'url' and 'timeout' properties var mycallback = function(result) { if (result.timeout == 'true') { // Try one more time response.object = y.rest(result.uri).get().response; } response.object = result; }

y.query
The syntax and usage for callbacks with y.query is nearly identical to that of y.rest. The callback function is passed as a parameter to y.query. The two properties timeout and query allow you to identify calls that have timed out and reuse the query. The code example below uses the callbackto keep track of the successful calls, logs the results, and saves the first returned response. // callback function logs the order in which url requests completed. var mycallback = function(result) { count++; y.log("Results received for query #" + count.toString() + ": " +result.query); if (first == null) { first = result.response; } } // create 4 asynchronous y.rest requests. y.query("select * from flickr.photos.search where has_geo='true' and text='san francisco'", mycallback); y.query("select * from feednormalizer where url='http://rss.news.yahoo.com/rss/topstories' and output='atom_1.0'", mycallback); y.query("select * from weather.forecast where location=90210", mycallback); y.query("select * from answers.search where query="cars" and category_id=2115500137 and type="resolved"", mycallback); This code example uses the timeout and query properties to log the queries that timed out. var mycallback = function(result) { if (result.timeout == 'true') { y.log("The following query did not return results: " + results.query; } } // Find photos in several cities y.query("select * from flickr.photos.search where has_geo='true' and text='san francisco'", mycallback) y.query("select * from flickr.photos.search where has_geo='true' and text='new york'", mycallback);

Yahoo! Developer Network

49

September 8, 2011

Executing JavaScript in Open Data Tables y.query("select * from flickr.photos.search where has_geo='true' and text='chicago'", mycallback);

Preventing Lossy JSON Results


When returning JSON responses from an Open Data Table, you should be aware that YQL will transform the JSON to XML before returning results to the client. If the client using your table requests that YQL return a JSON result, your original JSON response may be altered during the conversion from XML back to JSON, resulting in "lossy" JSON. To prevent "lossy" JSON, the client application using your table and requesting JSON results must add jsonCompat=new to the query string that is appended to one of the YQL Web service URLs. See JSONto-JSON Transformation [38] for details and usage examples. The YQL Query Parameters [3] also lists and describes the jsonCompat query parameter. When making REST calls to fetch JSON from an Open Data Table, you must append the jsonCompat=new to the query string to get "lossless" JSON as well. In the example table below, the REST call to the Gowalla API includes the jsonCompat=new query parameter to get "lossless" JSON. The client application using this table and requesting a JSON response must, however, also append jsonCompat=new to the YQL Web service URL to receive the "lossless" JSON. ... <bindings> <select> <urls> <url>http://api.gowalla.com/users/{id}</url> </urls> <inputs> <key id="id" type="xs:string" paramType="path" required="true"/> <key id="api_key" type="xs:string" paramType="variable" required="true"/> </inputs> <execute><![CDATA[ var req = request.header('X-Gowalla-API-Key', api_key).header('Accept', 'application/json'); response.object = req.query("jsonCompat","new").get().response; ]]> </execute> </select> </bindings> ...

Examples of Open Data Tables with JavaScript


The following Open Data Tables provide a few examples of YQL's abilities: Hello World Table [51] Yahoo! Messenger Status [51] OAuth Signed Request to Netflix [52]

Yahoo! Developer Network

50

September 8, 2011

Executing JavaScript in Open Data Tables Request for a Flickr frob [54] Celebrity Birthday Search using IMDB [55] Share Yahoo! Applications [58] CSS Selector for HTML [60]

Hello World Table


The following Open Data Table allows you to search a fictional table in which "a" is the path and "b" is the term. This table showcases the following: use of E4X to form the response <?xml version="1.0" encoding="UTF-8"?> <table xmlns="http://query.yahooapis.com/v1/schema/table.xsd"> <meta> <sampleQuery>select * from {table} where a='cat' and b='dog';</sampleQuery> </meta> <bindings> <select itemPath="" produces="XML"> <urls> <url>http://fake.url/{a}</url> </urls> <inputs> <key id='a' type='xs:string' paramType='path' required="true" /> <key id='b' type='xs:string' paramType='variable' required="true" /> </inputs> <execute><![CDATA[ // Your javascript goes here. We will run it on our servers response.object = <item> <url>{request.url}</url> <a>{a}</a> <b>{b}</b> </item>; ]]></execute> </select> </bindings> </table> Run this example in the YQL console.18

Yahoo! Messenger Status


The following Open Data Table allows you to see the status of a Yahoo! Messenger user.

18

http://bit.ly/eAvgR

Yahoo! Developer Network

51

September 8, 2011

Executing JavaScript in Open Data Tables The table showcases the following: use of JavaScript to check Yahoo! Messenger status use of E4X to form the response <?xml version="1.0" encoding="UTF-8"?> <table xmlns="http://query.yahooapis.com/v1/schema/table.xsd"> <meta> <sampleQuery>select * from {table} where u='sample_id';</sampleQuery> </meta> <bindings> <select itemPath="" produces="XML"> <urls> <url>http://opi.yahoo.com/online?m=t</url> </urls> <inputs> <key id='u' type='xs:string' paramType='query' required="true" /> </inputs> <execute><![CDATA[ //get plain text back from OPI endpoint rawStatus = request.get().response; //check if users is not offline if (!rawStatus.match("NOT ONLINE")) { status = "online"; } else { status = "offline"; } //return results as XML using e4x response.object = <messengerstatus> <yahoo_id>{u}</yahoo_id> <status>{status}</status> </messengerstatus>; ]]></execute> </select> </bindings> </table> Run this example in the YQL console.19

OAuth Signed Request to Netflix


The following Open Data Table allows you to make a two-legged OAuth signed request to Netflix. It performs a search on the Netflix catalog for specific titles20.
1 9

http://developer.yahoo.com/yql/console/?q=use%20%22http%3A%2F%2Fkid666.com%2Fyql%2Fymsg_opi.xml%22%20as%20ymsg.status%3B%20select%20*%20from%20ymsg.status%20where%20u%20%3D%20%22sample_id%22 20 http://developer.netflix.com/docs/REST_API_Reference#0_52696

Yahoo! Developer Network

52

September 8, 2011

Executing JavaScript in Open Data Tables This table showcases the following: access an authenticating API that requires signatures use an external JavaScript library <?xml version="1.0" encoding="UTF-8"?> <table xmlns="http://query.yahooapis.com/v1/schema/table.xsd" https="true"> <meta> <author>Paul Donnelly</author> <documentationURL>http://developer.netflix.com/docs/REST_API_Reference#0_52696</documentationURL> </meta> <bindings> <select itemPath="" produces="XML" > <urls> <url env="all">http://api.netflix.com/catalog/titles/</url> </urls> <paging model="offset"> <start id="start_index" default="0" /> <pagesize id="max_results" max="100" /> <total default="10" /> </paging> <inputs> <key id="term" type="xs:string" paramType="query" required="true" /> <key id="ck" type="xs:string" paramType="variable" required="true" /> <key id="cks" type="xs:string" paramType="variable" required="true" /> </inputs> <execute><![CDATA[ // Include the OAuth libraries from oauth.net y.include("http://oauth.googlecode.com/svn/code/javascript/oauth.js"); y.include("http://oauth.googlecode.com/svn/code/javascript/sha1.js"); // Collect all the parameters var encodedurl = request.url; var accessor = { consumerSecret: cks, tokenSecret: ""}; var message = { action: encodedurl, method: "GET", parameters: [["oauth_consumer_key",ck],["oauth_version","1.0"]]}; OAuth.setTimestampAndNonce(message); // Sign the request OAuth.SignatureMethod.sign(message, accessor); try { // get the content from service along with the OAuth header, and return the result back out response.object = request.contentType('application/xml').header("Authorization",

Yahoo! Developer Network

53

September 8, 2011

Executing JavaScript in Open Data Tables OAuth.getAuthorizationHeader("netflix.com", message.parameters)).get().response; } catch(err) { response.object = {'result':'failure', 'error': err}; } ]]></execute> </select> </bindings> </table> Run this example in the YQL console.21

Request for a Flickr frob


The following Open Data Table example returns the frob, which is analogous to the request token in OAuth. This table showcases the following: access an authenticating API that requires signatures use an external JavaScript library sign a request, then send the request using y.rest require the HTTPS protocol (since private keys are being transmitted) <?xml version="1.0" encoding="UTF-8" ?> // https="true" ensures that only HTTPs connections are allowed <table xmlns="http://query.yahooapis.com/v1/schema/table.xsd" https="true"> <meta> <sampleQuery> select * from {table}</sampleQuery> </meta> <bindings> <select itemPath="rsp" produces="XML"> <urls> <url>http://api.flickr.com/services/rest/</url> </urls> <inputs> <key id='method' type='xs:string' paramType='variable' const="true" default="flickr.auth.getFrob" /> <key id='api_key' type='xs:string' paramType='variable' required="true" /> <key id='secret' type='xs:string' paramType='variable' required="true" /> </inputs> <execute><![CDATA[ // Include the flickr signing library y.include("http://www.yqlblog.net/samples/flickr.js"); // GET the flickr result using a signed url var fs = new flickrSigner(api_key,secret);
21

http://bit.ly/7yNup

Yahoo! Developer Network

54

September 8, 2011

Executing JavaScript in Open Data Tables response.object = y.rest(fs.createUrl({method:method, format:""})).get().response(); ]]></execute> </select> </bindings> </table> Run this example in the YQL console.22

Celebrity Birthday Search using IMDB


The following Open Data Table retrieves information about celebrities whose birthday is today by default, or optionally, on a specific date. This table showcases the following: Creating an API/table from HTML data Mixing and matching Web service requests with HTML scraping Using E4X for creating new objects, filtering, and searching Parallel dispatching of query/REST calls Handling page parameters <?xml version="1.0" encoding="UTF-8" ?> <table xmlns="http://query.yahooapis.com/v1/schema/table.xsd"> <meta> <sampleQuery> select * from {table}</sampleQuery> </meta> <bindings> <select itemPath="birthdays.person" produces="XML"> <urls> <url></url> </urls> <paging model="offset"> <pagesize id="count" max="300" /> <total default="10" /> </paging> <inputs> <key id='date' type='xs:string' paramType='variable' /> </inputs> <execute><![CDATA[ //object to query imdb to extract bio info for a person var celebInfo = function(name,url) { this.url = url; this.name = name; var querystring = "select * from html where url = '"+url+"' and xpath=\"//div[@id='tn15']\""; this.query = y.query(querystring); }
22

http://bit.ly/18jOoM

Yahoo! Developer Network

55

September 8, 2011

Executing JavaScript in Open Data Tables

//actually extract the info and return an xml object celebInfo.prototype.getData=function() { default xml namespace =''; var d = this.query.results; var img = d..div.(@["id"]=="tn15lhs").div.a.img; var content = d..div.(@['id'] =="tn15content"); var bio = ""; //this is pretty hacky for each (var node in content.p) { if (node.text().toString().trim().length>100) { bio = node.*; break; } } var anchors = content.a; var bornInYear = null; var bornWhere = null; var diedInYear = null; var onThisDay = []; //TODO see if there is a wildcard way of pulling these out using e4x/xpath for each (var a in anchors) { var href = a.@['href'].toString(); if (href.indexOf("/BornInYear")==0) { bornInYear = a.toString().trim(); continue; } if (href.indexOf("/DiedInYear")==0) { diedInYear = a.toString().trim(); continue; } if (href.indexOf("/BornWhere")==0) { bornWhere = a.toString().trim(); continue; } if (href.indexOf("/OnThisDay")==0) { onThisDay.push(a.text().toString().trim()); continue; } } var bornDayMonth=null; var diedDayMonth=null; if (onThisDay.length>0) { bornDayMonth = onThisDay[0].replace(/^\s*(\d{1,2})[\s]+(\w+)\s*/,'$1 $2'); //tidy up whitespace around text if (diedInYear && onThisDay.length>1) { diedDayMonth= onThisDay[1].replace(/^\s*(\d{1,2})[\s]+(\w+)\s*/,'$1 $2'); //tidy up whitespace around text } } var url = this.url;

Yahoo! Developer Network

56

September 8, 2011

Executing JavaScript in Open Data Tables var name = this.name; var bornTime = null; if (bornDayMonth) { var daymonth = bornDayMonth.split(" "); bornTime=new Date(bornInYear,Date.getMonthFromString(daymonth[1]),parseInt(daymonth[0])).getTime()/1000; } var diedTime = null; if (diedDayMonth) { var daymonth = diedDayMonth.split(" "); diedTime=new Date(diedInYear,Date.getMonthFromString(daymonth[1]),parseInt(daymonth[0])).getTime()/1000; } var person = <person url={url}><name>{name}</name>{img}<born utime={bornTime}>{bornDayMonth} {bornInYear}</born></person>; if (diedTime) person.person+=<died utime={diedTime}>{diedDayMonth} {diedInYear}</died>; if (bio) person.person+=<bio>{bio}</bio>; return person; } //general useful routines String.prototype.trim =function() { return this.replace(/^[\s]*/,'').replace(/[\s]*$/,''); } Date.getMonthFromString = function(month) { return {'January':0, 'February':1, 'March':2, 'April':3, 'May':4, 'June':5, 'July':6, 'August':7, 'September':8, 'October':9, 'November':10, 'December':11}[month]; } Date.prototype.getMonthName = function() { return ['January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December'][this.getMonth()]; } //the main object that uses boss to get the list (also gets peoples "death" days too) celebSearch = function(when,start,count) { //search yahoo/boss using the current day and month only on bio pages on imdb var bornDayMonth = when.getDate()+" "+when.getMonthName(); var ud = Math.round(when.getTime()/1000); var search = 'site:www.imdb.com "Date of birth" "'+bornDayMonth+'" title:biography' var query = "select * from search.web("+start+","+count+") where query='"+search+"'"; var celebs = y.query(query).results; //go through each result and start to get the persons name and their imdb info page out var results = [];

Yahoo! Developer Network

57

September 8, 2011

Executing JavaScript in Open Data Tables default xml namespace ='http://www.inktomi.com/'; //make sure our e4x is in the right namespace. IMPORTANT for each (var celeb in celebs.result) { //discard any hits on the date of death that also match in our yahoo search //(this is going to hurt our paging) if (celeb["abstract"].toString().indexOf("<b>Date of Birth</b>. <b>"+bornDayMonth)<0) continue; var j = celeb.title.toString().indexOf("-"); //use text up to "dash" from title for name var name = celeb.title.toString().substring(0,j).trim(); //start parsing these entries by pulling from imdb directly results.push(new celebInfo(name,celeb.url)); } //loop through each imdb fetch result, and create the result object default xml namespace = ''; var data = <birthdays utime={ud} date={when} />; for each (var celeb in results) { data.birthdays+=celeb.getData(); } return data; } //run it for today if no date was provided var when = new Date(); if (date && date.length>0) { when = new Date(date); //TODO needs a well formed date including year } response.object = new celebSearch(when,0,count); ]]></execute> </select> </bindings> </table> Run this example in the YQL console.23

Shared Yahoo! Applications


The following Open Data Table provides a list of Yahoo! Applications that you and your friends have installed, indicating whether each app is installed exclusively by you, your friends, or both. This table showcases the following: complex E4X usage, including namespaces, filtering, searching, and creation authenticated calls to Yahoo! Social APIs using y.query setting a security level to user to force authenticated calls only

23

http://bit.ly/fV16L

Yahoo! Developer Network

58

September 8, 2011

Executing JavaScript in Open Data Tables optional variable that changes the function (searches on a specific friend) handling page parameters <?xml version="1.0" encoding="UTF-8" ?> <table xmlns="http://query.yahooapis.com/v1/schema/table.xsd" securityLevel="user"> <meta> <sampleQuery> select * from {table}</sampleQuery> </meta> <bindings> <select itemPath="root.install.app" produces="XML"> <urls> <url></url> </urls> <inputs> <key id='friendguid' type='xs:string' paramType='variable' /> </inputs> <execute><![CDATA[ function createInstallElement(update,type) { var bits = update.itemurl.toString().split("/"); var appid = bits[bits.length-2].substring(1);//get the appid from the install url var title = update.title.toString(); default xml namespace = ''; var el = <app who={type} id={appid}>{title}</app>; default xml namespace = 'http:://social.yahooapis.com/v1/updates/schema.rng'; return el; } default xml namespace = ''; var root = <install/>; //get my friends installs from updates var friendapp_installs = null; if (friendguid) { //only do deltas to this friend friendapp_installs = y.query('select title, itemtxt, itemurl from social.updates(1000) where guid=@guid and type="appInstall" | unique(field="itemtxt")',{guid:friendguid}); } else { //all friends friendapp_installs = y.query('select title, itemtxt, itemurl from social.updates(1000) where guid in (select guid from social.connections(0) where owner_guid=me) and type="appInstall" | unique(field="itemtxt")'); } //get my installs from updates var myapp_installs = y.query('select title, itemtxt, itemurl from social.updates(1000) where guid=me and type="appInstall" | unique(field="itemtxt")'); //we're going to keep a collection for each variant of the diff between my installs and my friend(s)

Yahoo! Developer Network

59

September 8, 2011

Executing JavaScript in Open Data Tables var myapp_installs = myapp_installs.results; var friendapp_installs = friendapp_installs.results; default xml namespace = 'http:://social.yahooapis.com/v1/updates/schema.rng'; for each (var myupdate in myapp_installs.update) { y.log("myupdate "+myupdate.localName()); //use e4x to search for matching node in friendapp with the same itemtxt (appid) var matching = friendapp_installs.update.(itemtxt==myupdate.itemtxt.toString()); if (matching.length()>0) { //found, we both have it root.install+=createInstallElement(myupdate,"shared"); //y.log("Found "+myupdate.title+" in both"); myupdate.@matched = true; matching.@matched = true; } else { //not in my friends apps, so add it to me only list //y.log("Found "+myupdate.title+" in mine only"); root.install+=createInstallElement(myupdate,"me"); myupdate.@matched = true; } } //anything left in the friends app list that doesnt have a "match" attribute is not installed by me for each (var friendupdate in friendapp_installs.update.(@matched!=true)) { root.install+=createInstallElement(friendupdate,"friend"); } //return the three sets of results default xml namespace = ''; response.object = <root>{root}</root>; ]]></execute> </select> </bindings> </table> Run this example in the YQL console.24

CSS Selector for HTML


The following Open Data Table allows you to filter HTML using CSS selectors. This table showcases the following: importing external JavaScript utility functions calling a YQL query within execute <?xml version="1.0" encoding="UTF-8" ?> <table xmlns="http://query.yahooapis.com/v1/schema/table.xsd"> <meta>
24

http://bit.ly/UeIuq

Yahoo! Developer Network

60

September 8, 2011

Executing JavaScript in Open Data Tables <sampleQuery>select * from {table} where url="www.yahoo.com" and css="#news a"</sampleQuery> </meta> <bindings> <select itemPath="" produces="XML"> <urls> <url></url> </urls> <inputs> <key id="url" type="xs:string" paramType="variable" required="true" /> <key id="css" type="xs:string" paramType="variable" /> </inputs> <execute><![CDATA[ //include css to xpath convert function y.include("http://james.padolsey.com/scripts/javascript/css2xpath.js"); var query = null; if (css) { var xpath = CSS2XPATH(css); y.log("xpath "+xpath); query = y.query("select * from html where url=@url and xpath=\""+xpath+"\"",{url:url}); } else { query = y.query("select * from html where url=@url",{url:url}); } response.object = query.results; ]]></execute> </select> </bindings> </table> Run this example in the YQL console.25

Execution Rate Limits


The following rate limits apply to executions within Open Data Tables: Item Total Units of Execution Total Time for Execution Total Stack Depth 50 million 30 seconds 100 levels Limit

Total Number of Concurrent YQL Queries 5 concurrent queries Total Number of Objects Created via new 1 million objects Total Number of Elements per E4X Object 1 million elements per E4X object What is a unit of execution?

25

http://bit.ly/IhF1b

Yahoo! Developer Network

61

September 8, 2011

Executing JavaScript in Open Data Tables A unit can be any usage of memory or instruction. For example, if a specific operation is only used twice within an execute script, that would sum up to 2 units: f(units) = f(operation1) + f(operation2)

Note
The total number of units allowed per operation can be lower than the maximum allowed if the script contains other operations which count towards the total units. The following unit costs apply toward execution rate limits: Unit y.query() 2000 units Methods of the y global object (such as y.log() and 1000 units y.rest()) String concatenation Operation of an object created via new Addition of an element Length of the string being concatenated (1 unit per character) 500 units per operation 50 units Cost

The following example calculates the number of units needed when adding two XML trees that each contain 10 elements: (10 elements + 10 elements) * 50 unit cost per element = 1000 units.

Yahoo! Developer Network

62

September 8, 2011

Chapter 3. Creating YQL Customized Functions


In this Chapter: Introduction [63] Types of Functions [63] How to Use a Customized Function [63] function Element [64] Stream Functions [64] Reduce Functions [66] Reference [68]

Introduction
YQL already has built-in functions that are used to decorate, modify, aggregate, or reduce items that are returned by a YQL query. Some of these built-in functions include the pipe operator (|), truncate, sanitize, and sort. YQL now lets developers use the execute element in the YQL Data Tables to create customized functions.

Types of Functions
Developers can create two types of functions using the execute element: Stream - This type of function gets called for each item at which point it can modify and return its modified version of the item. An example stream function might be called decorate because it would return each item with extra information. Reduce: This type of function gets called for each item which is collected, stored, and then finally returned as a reduced or aggregated response. The following are examples of built-in YQL reduce functions: sort, truncate, count

How to Use a Customized Function


Before you can use a customized function, you must include the table that defines the function with the key word use. Once you have access to the function though the table, you append the function to the query with the pipe operator that is used with built-in functions. The customized function, however, has to be called from the alias of the defining table and uses parenthesis to indicate it's a function. For example, if the decorate function is defined in the test.xml table, you would include the table with the below use statement: use 'http://<location-of-the-definition>/test.xml' as example;

Yahoo! Developer Network

63

September 8, 2011

Creating YQL Customized Functions

Because the alias for the table is example, you would call decorate from the alias and append it to the query with the pipe operate as shown here: select * from rss where url='http://rss.news.yahoo.com/rss/topstories' | example.decorate()

function Element
The YQL schema1 includes the new function element that lets table authors define functions within the execute element. Functions can use Java [23] or JavaScript as the execute language. The function element has a type attribute for specifying the type of function [63]. The value for the type attribute can be "stream" or "reduce". The function element also requires an input that allow the function to stream or reduce the response. That input is created with the pipe element, which is a sub-element of the inputs element. The pipe contains the current XML item which has passed the criteria of the YQL statement including any local filter applied to it. In this example function, the stream function uses one input pipe that passes the data to the JavaScript in the execute element. Notice that value of the id attribute of the pipe is also the variable name in the JavaScript: data ... <function name="decorate" type="stream"> <inputs> <pipe id="data" type="xs:string" paramType="variable"/> </inputs> <execute><![CDATA[ // Simply encapsulate the data XML object // into the results response.object = <results>{data}</results> ]]></execute> </function> ...

Stream Functions
To create a stream function, the open data table must have the following: a bindings element that has one or more function elements. For example: ... <bindings> <function name="decorate" type="stream"> ... </function> </bindings> ...

http://query.yahooapis.com/v1/schema/table.xsd

Yahoo! Developer Network

64

September 8, 2011

Creating YQL Customized Functions

function elements with the type attributes given the value "stream". For example: <function name="decorate" type="stream"> a pipe element with an id attribute that serves as one of the inputs. For example: ... <bindings> <function name="decorate" type="stream"> <inputs> <pipe id="data" type="xs:string" paramType="variable"/> </inputs> ... </function> </binding> ... Every stream function must have a pipe element or a runtime exception will be returned. a set response.object in the execute element to return to the caller. ... <bindings> <function name="decorate" type="stream"> <inputs> <pipe id="data" type="xs:string" paramType="variable"/> </inputs> <execute><![CDATA[ // Simply encapsulate the data xml object // into the results response.object = <results>{data}</results> ]]></execute> </function> </bindings> ...

Example Stream Function


The example YQL Data Table below defines the stream function decorate that simply wraps data in the <decorated> element. The following YQL statement shows you how to use the decorate function: use 'http://<location-of-the-definition>/decorate.xml' as example; select * from rss where url='http://rss.news.yahoo.com/rss/topstories' | example.decorate() YQL Data Table: decorate.xml <?xml version="1.0" encoding="UTF-8"?> <table xmlns="http://query.yahooapis.com/v1/schema/table.xsd" xmlns="http://query.yahooapis.com/v1/schema/table.xsd" securityLevel="any"> <meta> <author>nageshs</author> <description>Simple function call example</description> <documentationURL/>

Yahoo! Developer Network

65

September 8, 2011

Creating YQL Customized Functions

</meta> <bindings> <function name="decorate" type="stream"> <inputs> <pipe id="data" type="xs:string" paramType="variable"/> </inputs> <execute><![CDATA[ // Simply encapsulate the data xml object // into the decorated element response.object = <decorated>{data}</decorated> ]]></execute> </function> </bindings> </table>

Reduce Functions
To create a reduce function, the open data table must have the following: a bindings element that has one or more function elements. For example: ... <bindings> <function name="sort" type="reduce" itemPath="sorted.*"> ... </function> </bindings> ... function elements with the type attributes given the value "reduce"and an itemPath attribute in case the function returns multiple items. For example: <function name="sort" type="reduce" itemPath="sort.*"> a pipe element with an id attribute that serves as one of the inputs. Declare an itemPath in the case the function returns multiple items. For example: ... <bindings> <function name="sort" id="reduce" itemPath="sort.*"> <inputs> <pipe id="data" type="xs:string" paramType="variable"/> </inputs> ... </function> </binding> ... Every reduce function must have a pipe element or a runtime exception will be returned. The code in execute gets an implicit global variable called session that is a simple map for storing key-value pairs that are used by the application to construct the final response.

Yahoo! Developer Network

66

September 8, 2011

Creating YQL Customized Functions

... <binding> <function name="sort" type="reduce" itemPath="sorted.*"> <inputs> <pipe id="data" paramType="variable"/> <key id="key" type="xs:string" required="true" paramType="variable"/> </inputs> <execute><![CDATA[ // The function will be called with a non-null // data object until the entire stream is // exhausted. So a reduce function like sort // must collect all the data and *not* set // the response.object while there is data. if (data) { var value = y.xpath(data,"//" + key); // Also address 'content' access if (value == "") value = data.text(); if (value) { // Create the data structures that we // will populate during the collect operation if (!session.keys) session.keys = []; if (!session.data) session.data = {}; // Populate the current values session.keys.push(value); session.data[value] = data; } } ... ]]></execute> </function> </bindings> ... Reduce functions are called n+1 times where n is the number of items. After each item has been called once, the pipe id variable is set to "null", which indicates that the function can set the response.object. The else clause in the example code below sets the response.object after data is set to null. ... // The pipe id is 'data', which contains the items // returned from the YQL statement using the function. if (data) { var value = y.xpath(data,"//" + key); // Also address 'content' access if (value == "") value = data.text(); if (value) { // Create the data structures that we // will populate during the collect operation if (!session.keys) session.keys = []; if (!session.data) session.data = {}; // Populate the current values session.keys.push(value);

Yahoo! Developer Network

67

September 8, 2011

Creating YQL Customized Functions

session.data[value] = data; } } else { // Since data is null, the stream has // been exhausted and now we can perform the // reduce operation 'sort' in this case for each (var k in session.keys) y.log('key' + k); session.keys.sort(); var keys = session.keys; var result = <sorted></sorted>; for each(k in keys) { result.node += session.data[k]; } response.object = result; } ...

Reference
See the function element [37] in the Open Data Tables Reference2.

./yql-opentables-reference.html

Yahoo! Developer Network

68

September 8, 2011

Chapter 4. Using Hosted Storage with YQL


In this Chapter: Introduction [69] Storing New Records [70] Using YQL to Read, Update, and Delete Records [71] Using Records within YQL [72]

Introduction
YQL now provides two Open Data Tables, yql.storage and yql.storage.admin, that allow you to store and work with data using YQL itself. These default tables are available in the YQL console under the "yql" category. You interact with these Open Data Tables using the same keywords used with other YQL statements: namely SELECT, INSERT, UPDATE, and DELETE.

About YQL Hosted Storage


Data that you store for use with YQL is hosted in Yahoo!'s Sherpa cloud storage infrastructure. It is a scalable, elastic, and geographically distributed storage system used by many Yahoo! services. Benefits of using Sherpa include: low latency that can handle a large number of concurrent requests. automated load balancing and failover to reduce operational complexity. high availability and fault tolerance. For more information on Sherpa, refer to "Moving to the Cloud" on the Yahoo! Developer Network (YDN) Blog1.

Storage Limits and Requirements


The following storage limits and requirements apply to records stored using YQL: Size Limit: You can have up to 1000 records, with each record being up to 100KB. Retention Limit: Records not read, updated, or executed at least once every thirty days may be removed. Record Format: Records must be in a text-based format. Examples include JavaScript code, XML files, Open Data Tables, or YQL environment files. Authentication for New Records: All connections to yql.storage.admin must be authorized using two-legged OAuth. Alternatively, you can create new records using the YQL console. However, connections to yql.storage do not require authentication.
1

http://developer.yahoo.net/blog/archives/2009/06/sherpa.html

Yahoo! Developer Network

69

September 8, 2011

Using Hosted Storage with YQL

Storing New Records


You create new records in three ways using the yql.storage.admin Open Data Table: 1. value: Create a record and insert text into it. 2. url: Create a record and insert content from an existing URL. 3. name, url: Create a record with an execute key that you name and insert content from an existing URL.

Note
Any data you store within a record cannot exceed 100KB. Upon creating a new record, YQL responds with three access keys, each of which serves a different function, as seen in the following response snippet: ... <results> <inserted> <execute>store://35ad2c72-e353-41c4-9d21-4a7e5a1ec92</execute> <select>store://08fd2c74-d393-48c4-9ee1-3bf7e5a1ec92</select> <update>store://3cc85a99-6a89-4600-ade1-f7f83ecc4b83</update> </inserted> </results> ... There are three access keys associated with each record in storage, each starting with store://: execute: This access key allows the value of the record to be used in a YQL statement containing use statements or referring to env files. You can share this access key with other developers, who can use the Open Data Table or environment file but will be unable to read the table definition or environment details. select: This access key allows the value of the record to be used in YQL SELECT statements. update: This access key allows the value of the record to be used in YQL UPDATE and DELETE statements.

Note
Knowing the execute, select, or update access key is sufficient to perform those operations on the stored record. You should only share with others the access key that grants the desired permissions.

Storing a New Record using Text


To create a new text record in storage for YQL, use the following statement format: insert into yql.storage.admin (value) values ("example text content")

Yahoo! Developer Network

70

September 8, 2011

Using Hosted Storage with YQL

Storing a New Record using Data from an URL


To copy the contents of an URL, such as an environment file or Open Data Tables, into a new record for YQL, use the following statement format: insert into yql.storage.admin (url) values ("http://hostingdomain.com/mytable.xml")

Storing a New Named Record using Data from an URL


To copy the contents of an URL, such as an environment file or Open Data Tables, into a new record with a custom execute access key, use the following statement format: insert into yql.storage.admin (name,url) values ("newrecord","http://hostingdomain.com/mytable.xml") When you create a record using the above format, the execute access key uses the name (newrecord) and top-level domain of the URL (hostingdomain.com) that you supply, as seen in the following response snippet: ... <results> <inserted> <execute>store://hostingdomain.com/newrecord</execute> <select>store://08fd2c74-d393-48c4-9ee1-3bf7e5a1ec92</select> <update>store://3cc85a99-6a89-4600-ade1-f7f83ecc4b83</update> </inserted> </results> ...

Tip
Using the SET keyword in conjunction with a storage record, you can set and hide values, such as passwords, API keys, and other required values independently of YQL statements and API calls . For more information, refer to Setting Key Values for Open Data Tables [44].

Using YQL to Read, Update, and Delete Records


Once you create a record [70] using the yql.storage.admin Open Data Table, you can use the SELECT, UPDATE, and DELETE keywords with yql.storage to access, modify, or delete records, respectively.

Accessing Records using YQL


To read or access a record using YQL, use the following statement format that contains the SELECT keyword: select * from yql.storage where name="store://08fd2c74-d393-48c4-9ee13bf7e5a1ec92"

Yahoo! Developer Network

71

September 8, 2011

Using Hosted Storage with YQL

Use the select access key [70] that YQL provided to you upon creating the record. The YQL response contains a results element similar to the following: ... <results> <value>example test content</value> </results> ...

Deleting Records using YQL


Use the DELETE keyword in YQL to delete a record, similar to the following: delete from yql.storage where name="store://3cc85a99-6a89-4600-ade1f7f83ecc4b83" Use the update access key [70] that YQL provided to you upon creating the record. The YQL response contains a results element similar to the following: ... <results> <success>store://3cc85a99-6a89-4600-ade1-f7f83ecc4b83 deleted</success> </results> ...

Updating Records using YQL


Use the UPDATE keyword in YQL to modify a record, similar to the following: update yql.storage set value="new value" where name="store://3cc85a996a89-4600-ade1-f7f83ecc4b83" Use the update access key [70] that YQL provided to you upon creating the record. The YQL response contains a results element similar to the following: ... <results> <success>Updated store://3cc85a99-6a89-4600-ade1-f7f83ecc4b83</success> </results> ...

Note
If you create a named execute record [71], using a name and a url, you can only update the contents of this record to an URL on the same URL domain.

Using Records within YQL


If your record contains an environment file, an Open Data Table, or JavaScript, YQL can "execute" or run it in a read-only manner.

Yahoo! Developer Network

72

September 8, 2011

Using Hosted Storage with YQL

Using Hosted Environment Files


When a record contains a YQL environment file [42], use the corresponding execute access key [70] in YQL calls as you would any environment file. The following example uses URL encoding for the execute access key and appends it to the YQL console URL: http://developer.yahoo.com/yql/console?env=store%3A%2F%2Fopendatatables

Using Hosted YQL Open Data Tables


When a record contains an Open Data Table, use the corresponding execute access key [70] in YQL calls as you would any Open Data File. In the following example, the USE keyword invokes the access key, which is then used as mytable: use "store://35ad2c72-e353-41c4-9d21-4a7e5a1ec92" as mytable; select * from mytable;

Including Hosted JavaScript


When a record contains JavaScript, use the select access key [70] in YQL to include it in an Open Data Table.

Note
You do not use the execute access key [70] with JavaScript includes using y.include [21] because parsing JavaScript is done in the same manner as reading it. To do a JavaScript include, first insert the JavaScript into a record in the following manner: insert into yql.storage.admin (name, url) values ('testjs', 'http://javarants.com/yql/test.js') The response from YQL contains the following URLs: <inserted> <execute>store://javarants.com/testjs</execute> <select>store://f7c24314-b0fd-488b-a67c-1d711ca3bc21</select> <update>store://077b02c0-b6e8-4923-90a4-9b8096ccdaec</update> </inserted> Use the select URL in the select element [5] of the Open Data Table in the following manner: // contains: // var success = true; y.include('store://f7c24314-b0fd-488b-a67c-1d711ca3bc21'); response.object = <success>{success}</success>;

Yahoo! Developer Network

73

September 8, 2011

YQL for Yahoos

YQL for Yahoos


Abstract This part of the YQL Guide provides information and instruction for Yahoo! engineers using YQL. Yahoo! engineers can learn about internal-only features, learn how to create YQL tables for Yahoo! properties, and find a reference for the extended YQL schema. For general information about YQL, see Introduction YQL [1]. To learn how to use YQL statements, see Using YQL and Open Data Tables [1].

Table of Contents
1. Overview ........................................................................................................................ 1 Audience ................................................................................................................... 1 Why Use YQL? ........................................................................................................... 1 Features of Internal Open Data Tables ..................................................................... 1 YQL as Your Web Service ..................................................................................... 1 YQL Request Flow ...................................................................................................... 2 Authorization ...................................................................................................... 3 Request Parsing ................................................................................................... 3 2. Getting Started ................................................................................................................. 6 Requirements .............................................................................................................. 6 On-Boarding ............................................................................................................... 6 Accessing Internal Data Tables ....................................................................................... 7 Setup ................................................................................................................. 7 Example PHP Source Code ................................................................................... 8 Creating an External Open Data Table Containing Public Data ............................................. 9 Summary of Steps ................................................................................................ 9 Convert the Table Definition to a Core Table ........................................................... 10 Creating an External Open Data Table Containing Internal Data .......................................... 10 Summary of Steps .............................................................................................. 10 Get Paranoid Approval ........................................................................................ 11 Define a Table With Internal Data ......................................................................... 11 Internal YQL Endpoints for Development ............................................................... 12 Rate Limiting for Tables ...................................................................................... 13 Authenticated Data Access Through YQL ....................................................................... 16 Introduction ...................................................................................................... 16 Setup ............................................................................................................... 16 Examples ......................................................................................................... 16 3. Using YQL to Connect to a Database .................................................................................. 21 Requirements ............................................................................................................ 21 Making YQL Queries on the MySQL Database ................................................................ 21 Example YQL Statements and Responses ....................................................................... 22 4. Java Execute in Data Tables .............................................................................................. 23 Why Java Instead of JavaScript? ................................................................................... 27 Requirements ............................................................................................................ 28 Creating a YQL Data Table with Java Execute ................................................................ 29 5. Accessing Cookies from JavaScript Execute ........................................................................ 27 Use Cases ................................................................................................................. 27 Creating B-Cookies .................................................................................................... 27 Getting Cookies ......................................................................................................... 28 Setting Cookies ......................................................................................................... 28 Examples ................................................................................................................. 29 6. YQL Performance ........................................................................................................... 31 Latency versus Throughput .......................................................................................... 31 Monitoring YQL Performance ...................................................................................... 31 Caching and Storage ................................................................................................... 31 Optimizing YQL Calls ................................................................................................ 31 Make Parallel Calls ............................................................................................ 31 Use Timeouts .................................................................................................... 31 Check for Cached Queries .................................................................................. 32 Use XML ......................................................................................................... 32 Use YQL Caches and Storage ............................................................................... 32

Yahoo! Developer Network

iii

September 8, 2011

YQL for Yahoos

7. YQL Extended Schema .................................................................................................... 33 Introduction .............................................................................................................. 33 Reference ................................................................................................................. 33 table Element .................................................................................................... 33 javaExecute Element .......................................................................................... 34 connect Element ................................................................................................ 34 select | insert | update | delete Element .................................................................... 35 javaExecute Element (Within Bindings) ................................................................. 36 ratelimit Element ............................................................................................... 36 item Element ..................................................................................................... 36 function Element ............................................................................................... 37 Example Table Definitions ........................................................................................... 37 A. FAQ ............................................................................................................................ 38

Yahoo! Developer Network

iv

September 8, 2011

List of Figures
1.1. Request Flow Diagram ................................................................................................... 2

Yahoo! Developer Network

September 8, 2011

Chapter 1. Overview
This chapter explains how YQL can benefit Yahoos and describes certain features only available internally. For an overview of YQL, see Introducing YQL1.

Audience
This document is intended for internal Yahoo! developers. Typically, these developers are Yahoo! employees and contractors who work on Yahoo! properties and platforms. This is not intended for "external developers," that is, people who do not work for Yahoo! and do not have access to our internal data and environment within yahoo.com. More specifically, this chapter is intended for the following type of Yahoos: Yahoo! engineers wanting to make data available to other Yahoo! properties Yahoo! engineers wanting to access data from other Yahoo! properties

Why Use YQL?


YQL allows Yahoo! properties to supply data business and presentation logic separately from the data itself and create APIs that transparently merge data across several sources. In turn, Yahoo! engineers can use YQL tables to access data from other properties.

Features of Internal Open Data Tables


OAuth Scope Protection - we can protect your data by making your table accessible to only certain OAuth scopes (YQL acts as gatekeeper). You will still need to get a suitable scope added to YDN and membership so developers can choose the correct scope when creating their application. Rate Limiting - YQL tables can be plugged into the WS rate limiter, which will provide overall throughput limits as well as per IP, per CK, or per YCA limits. This will help you maintain maximum capacity and prevent network abuses that can happen when wrapping MySQL or VESPA services. MySQL Connectors to Execute Language - YQL exposes a MySQL connector object in the execute element of YQL Data Tables to get data from any database. Logging and Metric Information - YQL sends information to Yahoo! Web Analytics (YWA2) as your table gets used so you can track whats going on. YQL Customized Functions - YQL allows you to create your own functions to filter returned results.

YQL as Your Web Service


Many Yahoo! properties store their content in a mixture of HTTP accessible services like VESPA and databases such as MySQL. Moreover, most of the search-based Web services at Yahoo! (Local, Shopping, Answers, etc.) accept HTTP GET requests from their Web service implementation only to reroute them to VESPA to fetch data, strip and reformat fields, apply an XSLT if needed, and then return results.
1 2

http://developer.yahoo.com/yql/guide/yql_overview_guide.html http://twiki.corp.yahoo.com/view/SDSMain/IndextoolsIntegrationEngineering#YWA_Integrations_Deployments

Yahoo! Developer Network

September 8, 2011

Overview

YQL is middleware that provisions services, perform OAuth authorization, manage rate limits, and outputs results in several formats. Using YQL, Yahoo! properties can focus on the business logic of what and how the data should be provided and not how to implement HTTP accessible services. Using YQL Open Data Tables, Yahoo! properties create their own APIs that provide access to data through MySQL-like statements. Yahoo! properties can even create "write" Web services using the UPDATE, INSERT, and DELETE bindings.

YQL Request Flow


This chapter outlines the typical flow of a request within the YQL Engine. The various phases that the request undergoes and the outputs are also described. YQL requests go through the following phases: Authorization [3] Request Parsing [3] Request Execution [5] Response [5] The diagram below illustrates the request flow and some of the processes involved:

Figure 1.1. Request Flow Diagram

Yahoo! Developer Network

September 8, 2011

Overview

Authorization
YQL supports the following three authorization types: OAuth requests (either 2 legged or 3 legged) [3] Cookie+Crumb requests via YQL Console [3] Public endpoint requests [3]

OAuth Requests
Endpoint: /v1/yql OAuth requests are decoded by the mod_yahoo_webservice module, and the request is forwarded to the yjava_tomcat server. The Servlet Filter (called YahooAuthFilter) collects the following headers from the request and forward it on to the YQL Engine: x-yahoo-user-guid - denotes the GUID in case of a 3-legged OAuth request. Authorization

Cookie+Crumb Requests
Endpoint: /v1/yql/console This endpoint is for the YQL Console. A crumb is created and validated to prevent cross-site request forgeries, and the cookie is used to create an internal token using the function YOSTAuthProvider.createYOSTAuth().createInternalTokenWithYT(key, cookieJar). This internal token is then used as the Authorization header and forwarded along with the request to the YQL Engine.

Public Endpoint Requests


Endpoint: /v1/public/yql The public endpoint requires no authentication or authorization.

Common Headers
In addition to the Authorization and x-yahoo-user-guid headers, the following headers are also computed and forwarded to the YQL engine: YahooRemoteIP YahooRemoteIPSig Yahoo-App-Auth - the YCA certificate is forwarded to the YQL engine in case it needs to fetch any internal protected resources.

Request Parsing
Request parsing involves handling query string parameter or the parameters in the request body when HTTP POST, PUT, or DELETE calls are made.

Yahoo! Developer Network

September 8, 2011

Overview

The following happens during request parsing: Using the current request object, a new YQLPipeRequest is created. The value of the env parameter is used for the table environment. Setting table environment involves fetching the URL and adding all of the tables to the current tableScope. The q parameter is tranformed into a Pipe. This is invoked using YQLEngine.createPipe(yqlPipeRequest, tableScope).

Parse Request Into Expression Tree


The YQL lexer and YQL parser (YQLCompiler.parse()) is used to transform the request string into an abstract syntax tree (AST). This AST is then traversed in ASTVisitor to create a tree of Expression objects, and this expression tree is then typechecked by YQLCompiler.typeCheck() for syntactic errors.

Transform Expression Into Pipe


Because transforming the expression into a pipe is complicated, the process is described in the following ordered list: 1. The function YQLCompiler.toPipe() is called. 2. Any use table expressions and set value expressions are added to the environment. 3. Depending on the type of the root expression (YQL verb), one of the corresponding PipeCreator functions below is invoked: Select -> RESTPipeCreatorInsert Insert -> InsertPipeCreatorUpdate Delete -> DeletePipeCreator Describe -> DescribePipeCreatorValidate 4. Validate the request and ensure that I/U/D are called only using PUT/POST/DELETE or GET with a callback. 5. Call PipeCreator.process() to creates and returns a Pipe. 6. Get the ODT definition RESTTable from the table scope. 7. Traverse the expression tree and create corresponding Pipe objects such as Fetch, Filter, Union. 8. Optimize AND and OR trees by pulling OR to the highest level. 9. Find the matching endpoint from the ODT definition RESTTable. 10. Appropriate resources based on the YQL verb used. 11. Handle input types such as map, value, and key and any semantic errors such as missing required fields, duplicate fields, etc. Errors are reported in this phase. 12. Recursively process subselects to create the appropriate PipeCreator objects and process them.

Yahoo! Developer Network

September 8, 2011

Overview

Request Execution
The pipe created during the request parsing is executed in this phase. From the Pipe object, calls are made that traverse trees and invoke the leaves of the tree until a single Item object is returned. The Item objects from each tree are then collected and transformed into a single XML object.

Open Data Table (ODT) Execution


Before the request parsing phase, any symbols which point to Open Data Table URLs are populated into the environment. These symbols are resolved lazily and loaded into an instance of RESTTable. All the tables available to a particular request are passed to YQLCompiler in an object called RESTTableScopeRESTTable. These tables can either use standard schema (external) or extended schema (for internal use only). When the pipe is being exectuted at runtime, the following logic is applied: If the ODT has no execute element: table.binding.url is used as the uri template and populated with any keys, values or map input elements. url is fetched and the return is parsed into the format declared in the table (XML or JSON). If the ODT has an execute element: The Abuse tracker is checked to ensure that this particular ODT isn't part of the abusive list. The global and implicit objects (request, response, inputs, context) are created for the JavaScript. Each input is also declared as a global object in the JavaScript. If the ODT is internal, add the global object called internal. The JavaScript (Javascript) is executed. The response.object is coerced into XML or JSON and then itempath declared in the ODT is applied.

Response
The minimum (max-age) of all the resources is set as the Cache-Control header. Other headers such as Access-Control-Allow-Origin are also set up at this point. The XML object is then written to the output or transformed into JSON depending on the requested format.

Yahoo! Developer Network

September 8, 2011

Chapter 2. Getting Started


This section offers tutorials and practical information for Yahoos using YQL. The following topics are covered: Requirements [6] On-Boarding [6] Accessing Internal Open Data Tables [7] Creating an External Open Data Table Containing Public Data [9]

Requirements
You should be familiar with Yahoo!'s development environment and with the basic usage of YQL and YQL open data tables. In particular, you should know how to create a YQL data table. Those new to YQL should review the following YQL chapters: Using YQL and Open Data Tables1 Creating YQL Open Data Tables2

On-Boarding
Follow the steps below to expose your own tables, use your own tables, or use the YQL production instance: 1. Open an on-board ticket3 and state the following: requests per day / requests per second you expect for your solutionthis will help plan capacity. expected launch date for your product; please involve the YQL team early in planning stage. description of your use case and relevant information for your project. the colos from where you are planning to access YQL (confirm BCP). 2. Send an email to <yql-support@yahoo-inc.com> with the following information: brief description of your app/use case estimated average queries per second (QPS) / peak QPS time of slowest YQL query you expect to run 3. Develop your own tables using the production infrastructure (using your own environments and table definitions). Use <yql-discuss@yahoo-inc.com> for any questionsplease feel free to reach out with questions.

1 2

http://developer.yahoo.com/yql/guide/yql_users_guide.html http://developer.yahoo.com/yql/guide/yql_dev_guide.html 3 http://bug.corp.yahoo.com/enter_bug.cgi?product=YQL&component=On-boarding

Yahoo! Developer Network

September 8, 2011

Getting Started

4. (Optional) Schedule a quick meeting to see if the tables can be optimized. 5. Send a short email to a Paranoid stating that you know and approve of data being exposed via the tables. 6. Deploy your tables by doing one of the following: If the tables are globally available for external or internal use (but open to anyone), check the tables directly into our subversion repository. Your tables will be available after the next release push. If the tables are only going to be used by your application, leave them where they are (referenced by URL) or move them into the YQL storage system (which will obscure their definitions).

Accessing Internal Data Tables


This section shows how to set up and code a PHP program that queries an internal-only table requiring OAuth. For more information on this type of table, see Define a Table With Internal Data. [11] A Yahoo! property, for example, could use this PHP program to query a YQL table that contains internal-only data.

Setup
Prerequisite Tasks
1. Run some sample queries on the table with the YQL Console. If the table is still in development, run the YQL Console at one of the URIs listed in Internal YQL Console [12]. If the table is in production, run the production YQL Console4. 2. Find out the YQL service URI for the table. In the PHP source code, you will specify this URI for the $yqlEndPoint variable. If you are developing the table, select a URI from Internal YQL Endpoints for Development [12]. If the table is in production, use the production URI: http://query.yahooapis.com/v1/yql. 3. Make sure your development host is defined in a Yahoo! Role5. You must perform this step before installing the the yphp_yca package. In the PHP source code, you will specify the role for the $yacRole variable. 4. Get an internal consumer key6. In the PHP source code, you will specify the internal consumer key for the $consumerKey variable. 5. Run yinst to install the packages required for the PHP program. The following sections list the required packages.

Required yinst Packages for Development Hosts


yphp_ostauth yphp_yca
4 5

http://developer.yahoo.com/yql/console http://roles.corp.yahoo.com:9999/ui/home 6 http://twiki.corp.yahoo.com/view/YOS/NextGenAppID#Consumer_Keys

Yahoo! Developer Network

September 8, 2011

Getting Started

Select one of the following packages: wslogin_consumerkey_keys_alpha (if your host is in the yosnightly or yosstable environment) wslogin_consumerkey_keys_beta_gamma (if your host is in the yosint, yosqa, or yosgamma environment) Select one of the following packages: yck_data_alpha (if your host is in the yosnightly or yosstable environment) yck_data (if your host is in the yosint, yosqa, or yosgamma environment)

Required yinst Packages for Production Hosts


yphp_ostauth yphp_yca wslogin_consumerkey_keys_prod yck_data

Example PHP Source Code


The following program calls the YQL service to query an internal-only table. <?php function doYQLCall($yql) { $consumerKey = "<use internal consumer key>"; // In production, store the consumer key in a keyDB file; // do not hardcode it in the source code $ycaRole = "<use your own role>"; $yqlEndPoint = "http://query.yahooapis.com/v1/yql"; // Production //$yqlEndPoint = "http://yosgamma.pipes.yahoo.com/v1/yql"; // If using your dev box and not using yca proxy // (below), use this endpoint. $url = $yqlEndPoint.'?'.http_build_query(array('q' => $yql, 'format' => 'json', 'env' => 'store://datatables.org/alltableswithkeys')); $ost = ost_token_internal_create_yt_default($consumerKey); $s = yca_get_cert_once($ycaRole); $st_hdr = array(); $sth = "Yahoo-App-Auth:" . $s; array_push($st_hdr, $sth); $sth = "Authorization: YahooOAuthSession " . $ost; array_push($st_hdr, $sth); $c = curl_init(); curl_setopt($c, CURLOPT_URL, $url); curl_setopt($c, CURLOPT_PORT, 80);

Yahoo! Developer Network

September 8, 2011

Getting Started

// YCA proxying not needed for prod, // but needed for dev box // allows you to use production endpoint and // production consumerKey if using your devbox. // curl_setopt($c, CURLOPT_PROXY, 'http://yca-proxy.corp.yahoo.com:3128'); curl_setopt($c, CURLOPT_HTTPHEADER, $st_hdr); curl_setopt($c, CURLOPT_VERBOSE, 1); curl_setopt($c, CURLOPT_RETURNTRANSFER, 1); $d = curl_exec($c); $data = json_decode($d,1); return $data; } $data = doYQLCall('select * from social.connections where guid=me'); var_export($data); ?> See also Authenticated Data Access Through YQL [16].

Creating an External Open Data Table Containing Public Data


In this section, public data refers to Yahoo! data that is already available to end users or developers outside of Yahoo! (yahoo.com). The public data might be from the following types of sources: Web services on YDN CSV files such as historical quote data from Yahoo! Finance HTML pages to APIs (ScrAPI) such as shopping.y.jp

Summary of Steps
To create the YQL table, follow the same steps as an external developer, as described in Creating YQL Open Data Tables7. In summary, the steps are: 1. On your development server, create a table definition file that conforms to the "base" schema: http://query.yahooapis.com/v1/schema/table.xsd. 2. Test your YQL table by running queries in the production YQL Console8. 3. Next, test your table by writing a Web app that calls the production YQL service: http://query.yahooapis.com/v1/public/yql.

7 8

http://developer.yahoo.com/yql/guide/yql_dev_guide.html http://developer.yahoo.com/yql/console/

Yahoo! Developer Network

September 8, 2011

Getting Started

4. When your table is ready for production, perform one of the following tasks so that your table is available outside of yahoo.com: Recommended: Convert the Table Definition to Core Table. This option is recommended because core tables are the easiest to find. Share your table as a Community Open Data Table. Move the table definition file to a yahoo.com server that can be accessed outside of Yahoo!.

Convert the Table Definition to a Core Table


When the external table is ready for production, you should convert it to a core table. A "core table" is a YQL table that is built-in to YQL. In the YQL Console, core tables appear in the list under "Data Tables" after the user enters the "SHOW TABLES" YQL statement. Because they appear in YQL Console, core tables are more visible to external developers than Open Data Tables. To convert your YQL table to a core table: 1. Send an email to <yql-discuss@yahoo-inc.com> to contact the YQL team. 2. Check your table definition file into the SVN trunk for yql/conf/restdefs9. The YQL team monitors these check-ins. 3. The YQL team will let you know when your YQL table will become a core table in YQL production. 4. When your core table is in YQL production, try it out with the YQL Console and with an external Web app. Make sure that your Web app does not reference the table definition file you defined during development. (That is, the Web app does not specify the table definition file in a YQL "USE" statement.)

Creating an External Open Data Table Containing Internal Data


In this section, internal data refers to data that is accessible from a Web service on port 4080 in yahoo.com. Users and developers outside of the Yahoo! firewall cannot access data on port 4080. By creating a YQL table, you can expose internal Yahoo! data. For example, the Meme team exposes data from http://meme.yahoo.com:4080 in the Meme YQL Tables10. For security reasons, you may not expose data from corp.yahoo.com to users or developers outside of Yahoo!. For example, a YQL table may not expose Yahoo! Backyard search results from http://search.corp.yahoo.com. A production host in yahoo.com, including the YQL service, cannot access a host in corp.yahoo.com.

Summary of Steps
1. Get Paranoid Approval [11]. 2. Define a Table With Internal Data [11]. 3. To query your table, use one or more Internal YQL Endpoints for Development [12].
9

10

http://svn.corp.yahoo.com/view/yahoo/pipes/trunk/engine/yql/conf/restdefs/ http://developer.yahoo.com/meme/guide/meme-yql-tables.html

Yahoo! Developer Network

10

September 8, 2011

Getting Started

4. When your table is ready for production, Convert the Table Definition to a Core Table [10]. Your table cannot be accessed outside of the Yahoo! firewall until it becomes a core table.

Get Paranoid Approval


Paranoid approval is required before you can expose internal data to external developers or users. To get approval, contact the Paranoid representative for your team. Some internal data cannot be exposed in an external table because of legal agreements. For example, Yahoo! could have access to another company's data, but might not have permission to publish that data outside of Yahoo!.

Define a Table With Internal Data


To create a YQL table that contains Yahoo! internal data: 1. Create the table definition file on a gamma or beta server in your development environment. You may want to get started by copying and then modifying an existing table definition file from the SVN trunk for yql/conf/restdefs11. Here are a few sample files from that trunk: social.connections.xml12 - a simple file using auth="yahooOauth". social.notifications.xml13 - a table definition showing YQL Execute and with multiple bindings The table definition file must conform to the "extended" schema: http://query.yahooapis.com/v1/schema/internalTable.xsd. For more information, see YQL Extended Schema Reference. 2. In the table definition file, disable YQL's default proxy by setting useProxy="false" in each select/insert/update/delete element. For example: <i:select itemPath="" produces="XML" useProxy="false"> Alternatively, you may omit useProxy because its default value is false: <i:select itemPath="" produces="XML"> 3. Also in the table definition file, indicate the authorization required by the internal (4080) Web service by setting the auth attribute of the select/insert/update/delete elements. You must also set the securityLevel attribute of the table element (examples follow). Three-legged OAuth: If the internal Web service uses OAuth and requires user credentials, set securityLevel="user" and auth="yahooOauth". For example: <i:table . . . securityLevel="user"> . . . <i:select itemPath="" produces="XML" auth="yahooOauth"> Two-legged OAuth: If the internal Web service uses OAuth but does not require user credentials, set securityLevel="app" and auth="yahooOauth".

11 12

http://svn.corp.yahoo.com/view/yahoo/pipes/trunk/engine/yql/conf/restdefs/ http://svn.corp.yahoo.com/view/yahoo/pipes/trunk/engine/yql/conf/restdefs/social.connections.xml?revision=6791&view=markup 13 http://svn.corp.yahoo.com/view/yahoo/pipes/trunk/engine/yql/conf/restdefs/social.notifications.xml?view=markup

Yahoo! Developer Network

11

September 8, 2011

Getting Started

For example: <i:table . . . securityLevel="app"> . . . <i:select itemPath="" produces="XML" auth="yahooOauth"> Y and T only: If the internal Web service authenticates with Y and T cookies (and does not require OAuth) set securityLevel="user" and auth="yahooCookie". For example: <i:table . . . securityLevel="user"> . . . <i:select itemPath="" produces="XML" auth="yahooCookie"> For more information on the attributes related to authorization, see select | insert | update | delete Elements [35] and Backposting Yahoo! Cookies Using auth="yahooCookie" [19].

Internal YQL Endpoints for Development


Internal YQL Console
The URI for an internal instance of the YQL Console has the following syntax: http://{host}/v1/test/console.html Replace {host} with a value listed in Internal YQL Hosts [12]. For example: http://yosgamma.pipes.yahoo.com/v1/test/console.html The internal YQL Console requires a cookie for authentication, so you will be prompted to log in with your Yahoo! ID.

YQL Service for Tables Requiring OAuth


If the table requires OAuth authorization, the URI for the YQL service has the following syntax: http://{host}/v1/yql Replace {host} with a value listed in Internal YQL Hosts [12].

YQL Service for Tables Not Requiring OAuth


If the YQL table does not require credentials or OAuth, then the internal URI of the YQL service has the following syntax: http://{host}/v1/public/yql Replace {host} with a value listed in Internal YQL Hosts [12].

YQL Internal Hosts


The following table lists the hosts for development and testing:

Yahoo! Developer Network

12

September 8, 2011

Getting Started

Host staging.query.yahooapis.com

Description Open developer audience, stable pre-production, if you're writing a Y! app you must on-board to internal cluster [6]. Stable build for testing. Integration build, has latest features ("bleeding edge"). Also "bleeding edge".

gamma.yql.yahooapis.com yosint.pipes.yahoo.com:4080 yosbeta.pipes.yahoo.com:4080

y o s n i g h t l y . p i p e s . r e 1 . y a - Nightly build (even more "bleeding" edge than the preceding two). hoo.com:4080

Rate Limiting for Tables


YQL will integrate with the WS Rate Limiter to give table developers fine grained rate limiting. This is the YQL version of http://twiki.corp.yahoo.com/view/YOS/YQL_RateLimiting_Requirements

What Can be Limited?


Overall service limits/traffic to the table (and therefore the underlying service/data sources) Queries per day and/or queries per second (QPS) Progressive denial: favoring internal traffic over anonymous calls while maintaining QPS Per key limits YCA limits, including default for all YCA calls AppID/CK limits, including default for all AppID (OAuth) calls IP limits (for anonymous access), including default for all IP (anonymous) calls Soft and hard limits Soft limits continue to work, but hard limits stop traffic Alerting and messaging Emails when limits are hit Configuration and monitoring will take place using a dashboard created in the rate limiter.

How to Integrate WS Rate Limiting


In summary, property owners wanting to integrate WS Rate Limiting (RL) for their tables need to do the following: 1. Request a Rate Limiter ID by sending an email to <wsrl-eng@yahoo-inc.com>. The WS RL team will provision a dashboard that provides views of the number of requests and the types. 2. Include the ratelimit element in the extended schema. [?]

Yahoo! Developer Network

13

September 8, 2011

Getting Started

3. Set up Rate Limiter on integration/production with various limits and test that the rate limiter works as desired.

Include ratelimit Element in the Extended Schema


The URL or unique identifier for a rate-limited dashboard is provided in the table. Associated information such as owner, alert backyard ID, overall QPS/QPD is also provided in the dashboard. The provisioned rate limiter can be used for a group of tables at the discretion of the table owner. Add the ratelimit element with the URL or unique identifier to the extended schema of your table. The unique identifier "yahoo.music.id" is assigned to the id attribute of the ratelimit element in the example table below: <i:table xmlns:i="http://query.yahooapis.com/v1/schema/internalTable.xsd" xmlns="http://query.yahooapis.com/v1/schema/table.xsd" > <meta> <author>Yahoo! Inc.</author> <sampleQuery>select * from {table}</sampleQuery> </meta> <i:bindings> <i:select itemPath="" produces="XML" useProxy="false"> <urls> <url></url> </urls> <execute><![CDATA[ var auth = internal.auth.authorization; response.object = <item> <auth>{auth}</auth> <cookie>{y.context.auth.cookie}</cookie> <ck>{y.context.auth.ck}</ck> <guid>{y.context.auth.guid}</guid> <scope>{y.context.auth.oauthScopes}</scope> <appid>{y.context.auth.appId}</appid> </item> ]]> </execute> <i:ratelimit id="query.yahooapis.com/v11"/> </i:select> </i:bindings> </i:table>

Set Up a Rate Limiter for a YQL Table


Table owner first provisions a new rate limiter dashboard by configuring one. This can take place using a purpose built form via YQL that will talk to the Rate Limiter API to provision it and have the fields shown above as form fields. The dashboard will enable a property to whitelist or blacklist different keys (appid, yca, ip, etc.), change rates, move limits from hard to soft, etc. Each rate limiter contains three types of sub-rate limits: per YCA role per second or per day per AppId (OAuth) per second or per day

Yahoo! Developer Network

14

September 8, 2011

Getting Started

per IP (no Auth) per second or per day Each rate is enforced independently depending on how the table is accessed, with YCA limits being used before app ID limits, and app ID limits being used before IP limits. Thus, if no authentication is used, then the IP limit is used. If YCA is used, then the YCA limit is used, regardless of the IP or the internal ck/cks. In addition, the rate limiter will support overall service traffic shaping limits. The limits may be soft or hard. Soft limits cause emails to be sent to the owner of the dashboard, but are not actively enforced. Hard limits cause emails to be sent and prevent any call from occurring.

Progressive Denial
As the overall service threshold is hit for a table YQL/RL will start denying IP access, then app ID access and finally YCA access. Thus, less important traffic will be stopped first. In 3.2 Rate Limiter Release Rate Limiter will support three thresholds: low threshold (70%) - denies IP medium threshold (80%) - denies app ID high threshold (90%) - denies YCA The 3.2 Rate Limiter will not support alerts for soft limits. One of the actions associated with the thresholds shown above could be to deny the soft-limits clients, so at a high threshold you could choose to deny all soft clients. In 3.3 Rate Limiter Release The threshold for 3.3 is set at 85%, but is configurable. When the capacity goes beyond default threshold percentage, Rate Limiter will automatically start denying the IP, followed by the app ID, and finally by YCA, until the capacity reaches 85%. Automatic denial will be supported in 3.3, as will "soft limit" alerting (when the thresholds are hit they will trigger a warning but no action).

Rate Limiter Access from the YQL Engine


There are two ways of calling the rate limiter from the YQL engine or YQL table.

Automatic Limiting
The YQL Engine calls Rate Limiter with the unique ID found in the table definition. The YQL Engine goes through the following process: When accessing the table via YQL, the YQL engine uses the unique dashboard ID to run checkDecision as well as increment the access counts. YQL then writes increments in buckets for the dashboard ID: [tablename].[bindingtype]

Yahoo! Developer Network

15

September 8, 2011

Getting Started

Programmatic Limiting
YQL provides a custom API for non-trivial bucket updates. This API can be accessed from the execute element to handle any complex types of bucketing logic by deferring it to the table author. The y.ratelimit method in the code snippet below sets rate limiting based on the app ID and a parameter. ... <execute><![CDATA[ ... // Call Rate Limit to register and increment the // bucket values. // The ratelimit returns an exception if it // exceeds the specified limit as // detailed in the ratelimiter dashboard. y.ratelimit('appid_call_bucket', appId, 1); y.ratelimit('complex_parameter', param, 10); ]]!></execute>

Authenticated Data Access Through YQL


Introduction
YQL allows you to "backpost" or pass credentials through YQL to another Yahoo! Web Service. The credentials can consist of Y&T cookies, YCA certifications, or OAuth tokens and are backposted through the HTTP headers. This section contains code examples for backposting both OAuth and cookies. The examples are given in PHP or in executable JavaScript in a YQL Data Table.

Setup
See Setup [7] in Accessing Internal Data Tables.

Examples
Backposting OAuth Credentials
YQL handles and passes the OAuth tokens and the YCA certification for the caller to the intended Yahoo! Web service. You create the OAuth token with the Consumer Key and create the certificate with your YCA Role. For a code example that backposts OAuth credentials, see Accessing Internal Data Tables [7].

Backposting Y&T Cookie Credentials


This is an OO code example that takes the users Y&T cookies, creates an internal OAuth token, and then calls YQL. <?php /**

Yahoo! Developer Network

16

September 8, 2011

Getting Started

* @package YQL * @subpackage client * @author Dustin Whittle <dustin@yahoo-inc.com> * See recipes in Yahoo! Twiki: http://twiki.corp.yahoo.com/view/YOS/WSCookbookAuth * * Requires: * * 1) yinst install packages: yinst yphp_yca yphp_ostauth * 2) Internal Consumer Key * 3) YCA certificate * 4) Accessed through yapache with a user with valid y/t cookies * * * // alpha http://yosnightly.apps.yahoo.com:9999/creation_tool/getckey.php * * Backyard Login: dustin * Yahoo! ID: dustin.whittle * Yahoo! GUID: WYGRYNJNTQSQGKTNSLISLHMAWM * Consumer Key: dj0yJmk9bVkzTWkzdUV4UTF3JmQ9WVdrOUxXUjNZWEJ3Tm5FbWNHbzlNQ1o1UFRFLSZzPWNvbnN1bWVyc2VjcmV0Jng9MjM* * // prod http://ckey.apps.corp.yahoo.com:9999/creation_tool/getckey.php * * Backyard Login: dustin * Yahoo! ID: dustin.whittle * Yahoo! GUID: ECPZF7D765KTAXPDKWS7GE7CUU * Consumer Key: dj0yJmk9WGFnMFE5bzl5UWFhJmQ9WVdrOUxXUjNZWEJ3Tm5FbWNHbzlNQ1o1UFRFLSZzPWNvbnN1bWVyc2VjcmV0Jng9YmY* * */ $consumerKey = 'dj0yJmk9WGFnMFE5bzl5UWFhJmQ9WVdrOUxXUjNZWEJ3Tm5FbWNHbzlNQ1o1UFRFLSZzPWNvbnN1bWVyc2VjcmV0Jng9YmY-'; $applicationId = 'MYAPP.ALL'; //YOUR YCA ROLE $yql = new YQL($consumerKey, $applicationId); $data = $yql->query('SELECT * FROM social.connections WHERE guid=me'); var_dump($data); class YQL { const YQL_OAUTH_API = 'http://query.yahooapis.com/v1/yql'; public $consumerKey, $applicationId, $token, $certificate, $curl = null; public function __construct($consumerKey, $applicationId) { $this->initialize($consumerKey, $applicationId); } public function initialize($consumerKey, $applicationId) { // Create curl instance

Yahoo! Developer Network

17

September 8, 2011

Getting Started

$this->curl = curl_init(); $this->setConsumerKey($consumerKey); $this->setApplicationId($applicationId); } public function setConsumerKey($consumerKey) { $this->consumerKey = $consumerKey; $token = ost_token_internal_create_yt_default($consumerKey); $this->setToken($token); } public function getConsumerKey() { return $this->consumerKey; } public function setApplicationId($applicationId) { $this->applicationId = $applicationId; $this->setCertificate(yca_get_cert_once($applicationId)); } public function getApplicationId() { return $this->applicationId; } public function setToken($token) { $this->token = $token; } public function getToken() { return $this->token; } public function setCertificate($certificate) { $this->certificate = $certificate; } public function getCertificate() { return $this->certificate; } public function query($yql) { // YQL URL $url = self::YQL_OAUTH_API.'?'.http_build_query(array('q' => $yql, 'format' => 'json', 'env' => 'store://datatables.org/alltableswithkeys')); curl_setopt($this->curl, CURLOPT_URL, $url); // OAuth + YCA header curl_setopt($this->curl, CURLOPT_HTTPHEADER, array("Authorization: YahooOAuthSession {$this->token}","Yahoo-App-Auth:{$this->certificate}")); // Return HTTP response content curl_setopt($this->curl, CURLOPT_RETURNTRANSFER, true); // Do not include HTTP response headers curl_setopt($this->curl, CURLOPT_HEADER, false); $response = curl_exec($this->curl); $data = json_decode($response); return $data; // $this->log(sprintf('{CURL} Url: %s, Response: %s, Stats: %s. Time: %s', $url, $response, var_export(curl_getinfo($this->curl), true), time())); } public function log($message) { error_log($message);

Yahoo! Developer Network

18

September 8, 2011

Getting Started

} public function __destruct() { curl_close($this->curl); } }

Backposting Y&T Cookies from YQL Execute


If an HTTP request is made on a Yahoo! production server ({property}.yahoo.com), you access internal cookies with internal.request.cookies and then pass those cookie credentials to the Yahoo! production server by making a REST call with y.rest as seen in the code snippet of the YQL Data Table below. ... <execute><![CDATA[ var cookies = internal.request.cookies; var strCookie = ''; for (var i = 0; i < cookies.length; i++) { y.log(cookies[i].name); y.log(cookies[i].value); strCookie += cookies[i].name + '=' + cookies[i].value + ';'; if (i < cookies.length-1) strCookie += ' '; } // Now given this cookie send it to yahoo.com URL response.object = y.rest('http://...your.yahoo.com/somepage').header('Cookie', strCookie).get().response;]]> </execute> ...

Backposting Yahoo! Cookies Using auth="yahooCookie"


The YQL Extended Schema includes an auth attribute in the bindings for the data access statements such as select, insert, update, or delete. The auth attribute allows internal tables to declare that the authorization is based on cookies. When the value assigned to the auth attribute is "yahooCookie", the cookies are backposted with the request. The code example below uses the auth attribute for the select binding. <?xml version="1.0" encoding="UTF-8"?> <i:table xmlns:i="http://query.yahooapis.com/v1/schema/internalTable.xsd" xmlns="http://query.yahooapis.com/v1/schema/table.xsd" securityLevel="app"> <meta> <author>Yahoo! Inc.</author> <description>Social Updates table</description> <documentationURL> http://developer.yahoo.com/social/rest_api_guide/All-updates-resource.html</documentationURL> <sampleQuery>select * from {table} where guid=me</sampleQuery> </meta> <i:bindings> <i:select itemPath="" produces="JSON" auth="yahooCookie"> <urls>

Yahoo! Developer Network

19

September 8, 2011

Getting Started

<url>http://sandbox.bug.corp.yahoo.com:4080/api/1/wssid/json</url> </urls> <inputs> <key id="k" type="xs:string" paramType="query"/> </inputs> <execute><![CDATA[// Cookies from the original request are backposted to this request var data = request.get().response; response.object = <response>{data}</response>; ]]></execute> </i:select> </i:bindings> </i:table>

Yahoo! Developer Network

20

September 8, 2011

Chapter 3. Using YQL to Connect to a Database


Most properties across Yahoo! have their data available in a database (most commonly MySQL). Providing a way to expose the data directly, such as a Web service, makes the data both easy to access as well as compelling for properties to use. YQL makes data appear homogeneous, and the data can be mixed and matched with other services. To use the YQL to connect to a database, you make a REST call to one of the YQL internal hosts [12]. The REST call should include the q query string parameter with the YQL statement that connects to the database, provides database information, and has a subquery that queries data from the database table. To see how to form the YQL statement, see Making YQL Queries on the MySQL Database [21]. The YQL response will wrap the fields from the database table in row elements, making the response easy to parse and extract data. For an example response, see Example YQL Queries and Responses [22].

Requirements
You must be able to connect to a MySQL database given the URL to the database, the user name, and the password. You should be able to run SQL queries and get their response in either XML or JSON formats. Same Domain policy: The MySQL table can only be called from the same domain where the database resides. This is to prevent access to the database from malicious sources. The number of rows returned should be limited by the quota requirements followed by YQL. The connector should use a connection pool to ensure that the connection resource is reused.

Making YQL Queries on the MySQL Database


To use YQL to query a MySQL database, use the mysql.connector table in conjunction with the following keys: password - the password used to access the MySQL database. query - the YQL statement that queries the MySQL database. url - the URL to the MySQL database, such as jdbc:mysql://kindpleasure.corp.yahoo.com:3306/test user - the username for the database Using the mysql.connector table with the above keys, a sample query would have the following syntax: select * from mysql.connector where url='jdbc:mysql://{domain}.corp.yahoo.com:3306/{db_name}' and user='{db_username}' and passwd='{db_password}' and query="{YQL statement}"

Yahoo! Developer Network

21

September 8, 2011

Using YQL to Connect to a Database

Example YQL Statements and Responses


The following YQL statement shows the tables for the MySQL database test at the domain kindpleasure.corp.yahoo.com: select * from mysql.connector where url='jdbc:mysql://kindpleasure.corp.yahoo.com:3306/test' and user='myfoo' and passwd='some_pass' and query='show tables' The following YQL statement shows items from the shop table that have a price above 10 from the MySQL database test at the domain kindpleasure.corp.yahoo.com: select * from mysql.connector where url='jdbc:mysql://kindpleasure.corp.yahoo.com:3306/test' and user='myfoo' and passwd='some_pass' and query='select * from shop where price > 10' Below is the returned XML response: <?xml version="1.0" encoding="UTF-8"?> <query xmlns:yahoo="http://www.yahooapis.com/v1/base.rng" yahoo:count="2" yahoo:created="2010-01-11T07:58:39Z" yahoo:lang="en-US" yahoo:updated="2010-01-11T07:58:39Z" you=ts/sglpeyoc//l=lt+omqcnt+e+lD7b3ylAFFnlseo.h.mA02e%+ds%%mo2a+sdD7mps7nqr32ec*r+owrpc%+%" aor"t:ynhyisaoovy?se+fmyloeowru%%jcAs%%%kdeu.ryoc%36Fs2a+e32yo7nps%%sea%+duyD7lt+osphere312> h:ihp/oit.p.h.m1qqec*r+s.ncrher32d%mq322iparcpaoo33%tt7nurD7f%+daw32o_s2a+e%%se+fmh+e+i+E07 <diagnostics> <publiclyCallable>false</publiclyCallable> <user-time>135</user-time> <service-time>0</service-time> <build-version>4469</build-version> </diagnostics> <results> <row> <article>0002</article> <dealer>A</dealer> <price>10.99</price> </row> <row> <article>0004</article> <dealer>D</dealer> <price>19.95</price> </row> </results> </query>

Yahoo! Developer Network

22

September 8, 2011

Chapter 4. Java Execute in Data Tables


You can now execute Java instead of JavaScript from a YQL data table. This section discusses the advantages of using Java and provides an easy tutorial. Some YQL data tables are already using Java Execute, such as yahoo.dht, yql.query, yql.query.queries, and xslt.

Note
Any property using this service should consult with YQL team because there are implications and constraints that must be addressed to fully utilize this plugin model in the YQL environment.

Why Java Instead of JavaScript?


There is a strong desire to have a fully compiled version of Execute which would run much faster than the current JavaScript Execute. Even though internal tables are compiled, they are not stored as class files. There is an cost incurred to execute and translate the methods and arguments for each execute call. With a pure Java model, the compiled classes can be directly instantiated and run without incurring the JS runtime cost. Java inherently has capabilities that make it easier to do certain things. The Rhino JS runtime currently does not implement the E4X specification efficiently, leading to three times the number objects being generated for a single XPath. Other inefficiencies in the JS runtime can be totally bypassed if we have a pure Java runtime.

Requirements
javac 1.6+ Apache Maven publicly accessible server for storing jar

Creating a YQL Data Table with Java Execute


Set up the YQL development environment1 and follow the steps to create a simple YQL Data Table that implements Java Execute. 1. In a new directory, create a maven project: mvn archetype:create \ -DarchetypeGroupId=org.apache.maven.archetypes \ -DgroupId=com.example.javaexecute \ -DartifactId=jyql-execute

http://twiki.corp.yahoo.com/view/YOS/YQLDevEnv

Yahoo! Developer Network

23

September 8, 2011

Java Execute in Data Tables

2. Edit pom.xml and add the following dependency: <dependency> <groupId>com.yahoo.platforms.rsspipes.yql</groupId> <artifactId>yql-engine</artifactId> <version>1.1.3-SNAPSHOT</version> <scope>compile</scope> </dependency> 3. Create the class MyTable in the groupId package (com.example.javaexecute) using the code below: package com.example.javaexecute; import com.yahoo.platforms.yql.javaexecute.RESTRequest; import com.yahoo.platforms.yql.javaexecute.Y; import com.yahoo.platforms.yql.javaexecute.RESTResponse; import com.yahoo.platforms.yql.restdef.Response; import com.yahoo.platforms.yql.table.Input; import java.util.concurrent.ExecutionException; import java.util.concurrent.Future; import org.w3c.dom.Document; /*** Simple YQL Execute example*/ public class MyTable { /** * A simple select method which is referenced * from your YQL Open Data Table. It needs the request * and the response objects as parameters and * the Y object. Note that the orders of the keys * is not important. The java execute figures out the * correct object and passes the values. * Any input elements declared in the table are * injected into this code by looking at the input * annotations. In the following case,"@Input("mykey")" * maps to the actual declared in the input(s) section * of the document. * @param request * @param response * @param key */ public void select(RESTRequest request, Response response, Y y, @Input("mykey") String key) throws ExecutionException, InterruptedException { // Request is already populated with any URL // template attributes such as query parameters. Future<RESTResponse> resp = request.get(); // The URL is fetched in an executor asynchronously // so that the code is not blocked until a call to // Future.get() Object doc = resp.get().getResponse(); // The response is XML in this particular case: if (doc instanceof Document) { response.setObject(doc);

Yahoo! Developer Network

24

September 8, 2011

Java Execute in Data Tables

} // You can use XMLUtil.createDocumentBuilder(true).newDocument() // to create a new document to be returned if // necessary. Its the helper method to setup a // document. } } 4. Confirm you are using version 1.6 of the Java compiler that supports the latest features. $javac -version 5. Configure maven to use version 1.6 of the Java compiler by adding the following to pom.xml: <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <configuration> <source>1.6</source> <target>1.6</target> </configuration> </plugin> </plugins> </build> 6. Install the package. $mvn clean install 7. Create the jar and put it in on a server that can be publicly accessed. 8. Create the file mytable.xml for your table with code below. Make sure to replace the URL to the jar assigned to the src attribute of the javaExecute element with the URL to your own jar. <?xml version="1.0" encoding="UTF-8"?> <i:table xmlns:i="http://query.yahooapis.com/v1/schema/internalTable.xsd" xmlns="http://query.yahooapis.com/v1/schema/table.xsd"> <meta> <sampleQuery>select * from {table}</sampleQuery> </meta> <!-- Replace the URL to the jar that is assigned to the src attribute with the URL to your own jar. Be sure to end the URL with the "!" --> <i:javaExecute class="com.example.javaexecute.MyTable" src="jar:http://beta.mlk.re3.yahoo.com/restdefs/jyql-execute-1.0-SNAPSHOT.jar!/"/> <i:bindings> <i:select itemPath="" produces="XML"> <url>http://rss.news.yahoo.com/rss/topstories</url>

<urls>

Yahoo! Developer Network

25

September 8, 2011

Java Execute in Data Tables

</urls> <inputs> <key id="mykey" paramType="variable" /> </inputs> <i:javaExecute method="select" /> </i:select> </i:bindings> </i:table> Note that the URI pointing to the jar must end with a "!", so the value of the src attribute above is "jar:http://beta.mlk.re3.yahoo.com/restdefs/jyql-execute-1.0-SNAPSHOT.jar!/" 9. Place your YQL data table on a server that can be publicly accessed. 10. To use your table with the Java Execute, run the YQL statement below in the YQL Console2: use 'http://{your_domain}/mytable.xml' ; select * from mytable; To make your table public, see Creating an External Open Data Table Containing Public Data [9] and Creating an External Open Data Table Containing Internal Data [10].

http://developer.yahoo.com/yql/console/

Yahoo! Developer Network

26

September 8, 2011

Chapter 5. Accessing Cookies from JavaScript Execute


In a YQL table, engineers can use JavaScript in the execute element to create, get, and set B-Cookies1. YQL provides methods that are called from the internal global object [38] for working with cookies.

Use Cases
Three common use cases are listed below and are represented in the diagram that follows: An application not having a B-Cookie makes a request to YQL to get data. An application having a B-Cookie makes a request to YQL. An application not having a B-Cookie makes a request to UDA YQL that calls a beacon server.

Creating B-Cookies
To create a B-Cookie, use internal.yck.createBCookie() [41] in the execute element of a YQL table. The function returns an object containing cookie names and values. You need to cast the BCookie object as a string before you can set the cookie with internal.response.setHeader("SetCookie", bcookie.toString()). The example table below creates a B-Cookie and then passes it in the HTTP header: <?xml version="1.0"?> <i:table xmlns:i="http://query.yahooapis.com/v1/schema/internalTable.xsd" xmlns="http://query.yahooapis.com/v1/schema/table.xsd"> <i:bindings> <i:select produces="JSON" itemPath=""> <execute><![CDATA[ var bcookie = internal.yck.createBCookie(); internal.response.setHeader("Set-Cookie", bcookie.toString());]]>
1

http://twiki.corp.yahoo.com/view/SRelevance/Bcookies

Yahoo! Developer Network

27

September 8, 2011

Accessing Cookies from JavaScript Execute

</execute> </i:select> </i:bindings> </i:table>

Getting Cookies
Cookies are accessed using internal.request.cookies [39], which returns a reference to all of the cookies. To get a specific cookie value, you have to iterate through the cookie collection. The following table iterates through the collection of cookies to get the value of the count cookie. <i:table xmlns:i="http://query.yahooapis.com/v1/schema/internalTable.xsd" xmlns="http://query.yahooapis.com/v1/schema/table.xsd"> <i:bindings> <i:select produces="JSON" itemPath=""> <execute><![CDATA[ internal.setCookie("count",1,500); var get_cookie_value = function(cookie_name) { var cookies = internal.request.cookies; for(var i=cookies.length; i--;) { if(cookies[i].name == cookie_name){ return cookies[i].value; } } } var count = parseInt(get_cookie_value("count")) || 1; internal.setCookie("count",count+1,500);]]> </execute> </i:select> </i:bindings> </i:table>

Setting Cookies
To set cookies, you use internal.setCookie(cookie_name, cookie_value, expires) [38]. The following table sets the value for the cookie_origin cookie for 10 minutes. <?xml version="1.0" encoding="UTF-8"?> <i:table xmlns:i="http://query.yahooapis.com/v1/schema/internalTable.xsd" xmlns="http://query.yahooapis.com/v1/schema/table.xsd"> <meta> <sampleQuery>select * from {table}</sampleQuery> </meta> <i:bindings> <i:select itemPath="" produces="XML"> <execute> <![CDATA[ internal.setCookie('cookie_origin', 'YQL', 600); // View your cookies y.log(internal.auth.cookie); </execute>

Yahoo! Developer Network

28

September 8, 2011

Accessing Cookies from JavaScript Execute </i:select> </i:bindings> </i:table>

Examples
This example uses several methods and properties from the internal object to manipulate cookies. Using internal.auth.cookie, the entire cookie collection is logged. To view the cookies as keyvalue pairs, the cookie collection is obtained with internal.request.cookies and then iterated through. Finally, the code checks to see if the HTTP header contains "Set-Cookie" before using internal.response.setHeader to set the value of another HTTP header. <?xml version="1.0"?> <i:table xmlns:i="http://query.yahooapis.com/v1/schema/internalTable.xsd" xmlns="http://query.yahooapis.com/v1/schema/table.xsd"> <i:bindings> <i:select produces="JSON" itemPath=""> <execute><![CDATA[ y.log(internal.auth.cookie); var cookies = internal.request.cookies; var strCookie = ''; for(var i=0; i<cookies.length; i++) { strCookie += cookies[i].name + '=' + cookies[i].value + ';'; if(i<cookies.length-1) strCookie += ' '; } y.log(strCookie); if(internal.response.containsHeader('Set-Cookie') == true) { internal.response.setHeader('jim', 'bob'); }]]> </execute> </i:select> </i:bindings> </i:table> This example table shows how to set cookie values with internal.setCookie. Because internal.setCookie doesn't return a value, we suggest you use a try-catch block when setting cookie values to confirm that the cookies have been set. <?xml version="1.0" encoding="UTF-8"?> <i:table xmlns:i="http://query.yahooapis.com/v1/schema/internalTable.xsd" xmlns="http://query.yahooapis.com/v1/schema/table.xsd"> <meta> <sampleQuery>select * from {table}</sampleQuery> </meta> <i:bindings> <i:select itemPath="" produces="XML"> <execute> <![CDATA[ try { internal.setCookie('f', 'bar', 600); internal.setCookie('j', 'doe', 600); response.object = 'hello world'; } catch(err) { y.log('exception caught'); y.log(err);

Yahoo! Developer Network

29

September 8, 2011

Accessing Cookies from JavaScript Execute }]]> </execute> </i:select> </i:bindings> </i:table>

Yahoo! Developer Network

30

September 8, 2011

Chapter 6. YQL Performance


Latency versus Throughput Monitoring YQL Performance Caching and Storage Optimizing YQL Calls
This section shows you several methods to optimize calls to the YQL Web service. You can also view the Performance and YQL presentation1.

Make Parallel Calls


The JavaScript methods y.rest(url) and y.query(statement) available in the execute element in Open Data Tables are executed as futures. This means that the calls made by these methods are returned immediately and your JavaScript continues to run while the network call is made in the background. Only when you ask for the response or results objects does YQL block (if needed) on the network call. YQL will execute a maximum of 5 concurrent requests for every query. The following code example makes parallel calls and provides inline comments to annotate the process: // 1. Make three parallel calls var one = y.query("select * from long query"); var two = y.query("select * from another query"); var three = y.rest("http://somewhere else"); // 2. The three network calls are in process // 3. YQL blocks the return for 'one'; // The network calls to 'two' and 'three' continue var oneResult = one.results; // 4. YQL blocks on the return for 'two'; // The network calls to 'three' continues var twoResult = two.results; // 5. YQL blocks on the return for 'three' var threeResult = three.response;

Use Timeouts
By default, YQL will wait 30 seconds for a network response. If you have an SLA, you may need to stop a network call earlier. Using timeouts is the best way to control the time needed for a network call. YQL allows you to specify timeouts for sub-queries and REST calls.
1

http://twiki.corp.yahoo.com/view/YOS/YqlPerformanceAndCaching?slideshow=on;skin=print#GoSlide1

Yahoo! Developer Network

31

September 8, 2011

YQL Performance

The following sets a 5-second timeout on the query: var q = y.query("select * from foo",{},5); In this code snippet, a 5-second timeout is set on the REST call: var r = y.rest("http://api.example.com").timeout(500).get();

Check for Cached Queries


Because y.query doesn't make a network request to YQL, it cannot use YQL's front-facing HTTP caching logic. You can, however, easily use y.cache to wrap calls if they are expensive and the data results change infrequently. In the following code example, the YQL cache is checked before the query is made. The returned results are then cached. var query="select * from foo where cat='dog'"; // Try to get the cached results (and cache key) var result = y.cache.get(query); // See if we have a cached result if (!result) { // If not found in memcache, make a query var result = y.query(query).results; // Cache the results for 100 ms y.cache.put(query,result,100); }

Use XML
YQL uses XML in its internal engine, so consuming or producing other formats, such as JSON, is more expensive than XML. If in doubt, request or create XML responses.

Use YQL Caches and Storage


Serving from an HTTP cache anywhere in YQL's architecture is a good thing. By default, YQL sets the maxage HTTP header to be the fastest expiring header for all of the HTTP resources the query in YQL accessed. For example, if YQL accessed three feeds with maxages (or expires) of 10s, 100s and 1000s, then the maxage header set by YQL is 10s. To make your code and application run faster, use any of the following caches: _maxage=NNN as a CLIENT to over-ride the default maxage header returned by YQL. response.maxAge=NNN in the execute element of the ODT to set the maxage response explicitly. y.cache in the execute element of the ODT to store items in memcache or cache results. yahoo.dht table to access and store Sherpa records. y.rest(...).header("Cache-Control: max-stale=300") to get stale contents from the squid cache faster (while causing a stale while re-validate request). By default, YQL always ask for 300s (5 minutes) as the max stale from the from squid cache.

Yahoo! Developer Network

32

September 8, 2011

Chapter 7. YQL Extended Schema


Introduction
YQL table definitions are based on the following XSD files: base schema - Defines the rules for YQL tables created by both external and internal developers: http://query.yahooapis.com/v1/schema/table.xsd1 The elements for the base schema are documented in the Open Data Tables Reference2.

Reference
The YQL extended schema defines the following elements: table javaExecute connects connect whitelist item binding function select | insert | update | delete javaExecute ratelimit

table Element
The root element for the document that defines a YQL table. For more attributes, see the table element3 (base schema) in the YQL Guide. Full Path: root element Example: <i:table xmlns:i="http://query.yahooapis.com/v1/schema/internalTable.xsd" xmlns="http://query.yahooapis.com/v1/schema/table.xsd" securityLevel="user" yahooInternal="true">
1 2

http://query.yahooapis.com/v1/schema/table.xsd http://developer.yahoo.com/yql/guide/yql-opentables-reference.html 3 http://developer.yahoo.com/yql/guide/yql-opentables-reference.html#yql-opentables-tables-element

Yahoo! Developer Network

33

September 8, 2011

YQL Extended Schema

Attributes xmls

Value <i:table xmlns:i="http://query.yahooapis.com/v1/schema/internalTable.xsd" xmlns="http://query.yahooapis.com/v1/schema/table.xsd" securityLevel="user" yahooInternal="true">

Description The XML namespace, identified by the i: prefix and the XML schema file name. In the YQL docs, internalTable.xsd is referred to as the "extended schema."

yahooInt- boolean ernal hidden boolean

Setting to "true" makes the table available only within Yahoo!. Default is "false". Setting to "true" restricts the presence of this table in the "show tables" command. Default is "false".

javaExecute Element
Additional information on how the YQL Service connects to the URL within a named environment. Full Path: table/javaExecute Example: <i:javaExecute class="com.example.javaexecute.MyTable" src="jar:http://beta.mlk.re3.yahoo.com/restdefs/jyql-execute-1.0-SNAPSHOT.jar!/"/> Attribute Type Description class src string The Java class to instantiate. Example: com.yahoo.local.mytable URI The URI of the Java JAR file that contains the Java code needed for the table. The format of the URI is: jar:url! (note that the trailing bang is required). For example, to specify a JAR file named "getstuff.jar", the value of src should be jar:http://foo.com/getstuff.jar!

connect Element
Additional information on how the YQL Service connects to the URL within a named environment. Full Path: table/connects/connect Example: <i:connects> <i:connect env="prod" yca="yahoo.pipes.yql.yosprod" /> <i:connect env="gamma" yca="yahoo.pipes.yql.yosgamma" /> <i:connect env="beta" yca="yahoo.pipes.yql.yosbeta" /> <i:connect env="int" yca="yahoo.pipes.yql.yosint" /> <i:connect env="dev" yca="yahoo.pipes.yql.yosnightly" /> </i:connects>

Yahoo! Developer Network

34

September 8, 2011

YQL Extended Schema

Attribute Type Description env yca string The named environment. When the YQL Service is installed with yinst, the env is assigned. string The name of the YCA certificate for the environment. At runtime, YQL gets the certificate value from the certificate db, and then injects the certificate into the HTTP header YahooAppAuth. string The URI of a proxy server. Overrides the default proxy that YQL normally uses.

proxy

select | insert | update | delete Element


The bindings for the data access statements such as SELECT. See also the select... elements4 (base schema) in the YQL Guide. Full Path: table/bindings/ select | insert | update | delete Example: <i:select itemPath="" produces="JSON" auth="yahooOauth" useProxy="false">

Attribute useProxy

Type boolean

Description If useProxy is "true", then use the proxy indicated by the proxy attribute of the connect element. If useProxy is "false" and the proxy attribute is not specified for the connect element, then do not use a proxy. The default value of useProxy is "false". The host name that the user will see in the diagnostics. Specify displayHost to hide the actual host name. For example, if the host name is "foo.spq.yahoo.com/bar" and you do not want this host name to appear in the diagnostics, you could set displayHost to "sherpa". In this case, "sherpa/" will appear in the diagnostics.

displayHost URI

auth

enumertion The type of authorization or authentication. Allowed values: yahooOauth - Indicates that the backend Web service for the table requires OAuth authorization. yahooCookie - Indicates that the backend Web service for the table is protected by Y and T cookies, but not by OAuth. If the request to YQL is using 3-legged OAuth, then the YQL service will pass the cookie from the request to the backend Web service. If the request contains a Y and T cookies, those cookies are passed on to the backend Web service.

authScope

string

See authScope [35].

authScope Attribute
The authScope attribute is optional. You should set authScope if you want scope checking performed, but the Web service accessed by the table does not perform scope checking. The value of authScope is a comma-separated of list of authorization scopes (permissions) that control access to Yahoo! data.

http://developer.yahoo.com/yql/guide/yql-opentables-reference.html#yql-opentables-select

Yahoo! Developer Network

35

September 8, 2011

YQL Extended Schema

For example: authScope='sdps-r,deli-w,mail-w' If authScope is set, scope checking is performed if the Web service accessed by the table uses threelegged OAuth. The settings in authScope must match the scope settings in the OAuth session token of the request. Otherwise, an error occurs and access is denied. If the Web service accessed by the table is protected only by Y and T cookies, then authScope is ignored and scope checking is not performed. If the OAuth token of the request to YQL is internal (generated by a Yahoo! property), then authScope is ignored and scope checking is not performed.

javaExecute Element (Within Bindings)


The Java method to execute for the select, insert, update, or delete. This Java method belongs to the class specified by the table/javaExecute [34] element. Full Path: table/bindings/ select | insert | update | delete /javaExecute Example: <i:javaExecute method="getStuff" /> Attribute Type Description method string The name of the Java method to execute.

ratelimit Element
Impose rate limiting on table queries. For more information, see YQL Rate Limiting. Full Path: table/bindings/ select | insert | update | delete /ratelimit Example: <i:ratelimit id="query.yahooapis.com/v1"/> Attribute Type Description id string The identifier for a specific rate limit setting. To get the identifier, you must register with the rate limiting dashboard. The rate limiting project is called "Hawk-I".

item Element
TBD short desc Full Path: table/whitelist/item Example: TBD Attribute Type Description id string TBD

Yahoo! Developer Network

36

September 8, 2011

YQL Extended Schema

Attribute Type Description type string TBD

function Element
A stored procedure to bind and run, similar to the built-in Sort and Other Functions5. For more information on the function element, see the YQL Function Implementation. Full Path: table/bindings/function Example: <table> <bindings> <function name="concat" /> <inputs> ... </inputs> <execute> ... </execute> </function> ... </bindings> <table> Attribute Type name type string Description The name of the function.

enumeration The type of function. Allowed values: stream - (Default) This type of function gets called for each item at which point it can modify and return the modified version of the item. Example: Decorate an item with extra information. reduce - This type of function gets called for each item which is collected/stored and then finally, gets one last call to return the reduced or aggregated response. Examples: sort, truncate, count.

Example Table Definitions


The following table definition files conform to the extended schema: social.connections.xml6 - a simple file using auth="yahooOauth". social.notifications.xml7 - a table definition showing YQL Execute and with multiple bindings. For more examples, see the files in the SVN trunk for yql/conf/restdefs8.
5 6 7

http://developer.yahoo.com/yql/guide/sorting.html http://svn.corp.yahoo.com/view/yahoo/pipes/trunk/engine/yql/conf/restdefs/social.connections.xml?revision=6791&view=markup http://svn.corp.yahoo.com/view/yahoo/pipes/trunk/engine/yql/conf/restdefs/social.notifications.xml?view=markup 8 http://svn.corp.yahoo.com/view/yahoo/pipes/trunk/engine/yql/conf/restdefs/

Yahoo! Developer Network

37

September 8, 2011

Appendix A. FAQ
A.1. What data can YQL abstract or connect to? YQL just needs to get data from somewhere (not just a web service), which includes VESPA, Sherpa, etc. So if you had an internal service today you wanted an external version of, YQL can provide the access layer and data presentation logic for you. Of if you didnt have a web service at all but had vespa you can externalize that via YQL. Or if you had a CSV style service (finance) and wanted to create a rich XML version, you could do that (and we did). Or if you have an internal web service you want to simply make easier to use (add JSONP etc to your XML), you can create a table for it and only make it available to other yahoo services. Etc. A.2. Who is using YQL today? See the YQL Customer Pipeline1 for the list of properties using YQL. A.3. Where are the YQL engines across the company? YQL currently is running on three engine machines in the following five colos: SP1, MUD, RE4, CH1 and SG1. This currently gives us a ~1200 QPS capacity. We are contemplating getting out of MUD in order to increase hardware more easily elsewhere. A.4. Does YQL have mechanisms for caching and storing data? YQL has access to four different types of cache/storage: 1. Forward facing: YTS (between YQL and developer/property) to cache responses from YQL 2. Backward facing: Squid via httproxy (between YQL and external non-Yahoo! data sources) force caches all content unless "private" 3. Memory caching: memcached accessible inside execute as y.cache single logical memory cache across all machines in a colo 4. DHT: Sherpa2 YQL provides full access to Sherpa tables via the yahoo.dht table A.5. Can YQL access resources across colos? YQL can access any resource across colos on port 4080. Accessing other resources, such asVESPA on port 5810, can be done by opening ACLs between YQL and the servers. A.6.
1 2

Are there any abuse policies or rate limiting that protect my service?

http://twiki.corp.yahoo.com/view/YOS/YqlCustomers http://twiki.corp.yahoo.com/view/YDHT/WebHome

Yahoo! Developer Network

38

September 8, 2011

FAQ

We are finishing up integrating the WS Rate Limiter which will provide automatic rate limiting dashboards and control for any table in YQL. A.7. Can the usage of YQL tables be tracked? We track all requests per table using YWA. Until YWA provides a programmatic API to get these out on a per-table basis you can use our YWA account. Send a request to use our YWA account to <yql-discuss@yahoo-inc.com>. A.8. Is there access control for tables? Yes, access to YQL tables limited to the following: only 2-legged OAuth only 3-legged OAuth public In effect, YQL takes care of the gatekeeping to your tables. In addition, YQL tables can be made only available to certain OAuth scopes. Your service does not need to use/provide OAuth to make use the OAuth protection. For example if you want to use a Y&T protected service you can backpost the Y&T from the OAuth request in your table. A.9. How can my data service authenticate with YQL? YQL can backpost OAuth tokens or Y&T cookies, or anything else provided in the request. YQL can also use any YCA role associated with our hosts (or none). A.10. Is there an on-boarding process? YQL is mostly self-serve. You can do all the development of tables and even use them without ever needing to talk to us, but Getting Started will show you how to create tables, optimize your calls, create functions, and more. For low QPS capacities (<50qps) there should be no problems using our shared production instance. If you are Metro/Mail and want several thousand QPS, then installing your own YQL instances is probably a better option. A.11. How to capture your YQL request? Sometimes things don't work as we expect, if you feel like there is an issue with YQL itself please try and capture a reproducible test case through tcpdump and retain the full HTTP request and response of the specific test case in question. Capture the data as ASCII to the console: $ sudo tcpdump -s 0 -A Or capture the raw binary data: $ sudo tcpdump -s 0 -w your_file_name_here A.12. How do I get information about YQL releases?

Yahoo! Developer Network

39

September 8, 2011

FAQ

YQL pushes candidates to our staging server, staging.query.yahooapis.com, approximately one week before being pushed to the rest of our production hosts. An email is sent to <YQL-announce@yahoo-inc.com> letting you know that a new release is available for regression testing. If you need more time, make your requests through <yql-discuss@yahoo-inc.com>. A.13. Why does Yahoo! Finance use community tables ('yahoo.finance.*')? Shouldn't their tables be part of the YQL Console's default set? Finance data is not an official API. Finance is unsupported, and the company is turning a blind eye to people using the data. It's definitely not 'sanctioned'. As a rule of thumb though, if the table is completely relying on public APIs and doesn't need any user-identifiable information (via Yahoo! OAuth), then it may be best to add it as a community table. The Yahoo! Finance tabales fall into that category. As an added benefit, being a community table lets other publishers see how to create open data tables. A.14. How do I shorten an URL with YQL? The yahoo.y.ahoo.it table uses bit.ly to shorten URLs. To get a yahoo.y.ahoo.it account, please contact Jeanne Moeschler3. The table is handled by YQL. More info on this service is available at Yahoo It4. A.15. How do I get the original URL for a yahoo.y.ahoo.it URL? Use a YQL statement with the following synatx: select * from yahoo.shortener where url='http://y.ahoo.it/ATDvNGdZ5 A.16. Is there a way of knowing the environment we load with the env parameter or the list of environments that we are able to load using the y.env function? TBD A.17. Is it possible to get the totalhits field (number of search results) from the search.web table in YQL? The field is documented in BOSS API Guide6, but I don't see it returned in the YQL response7. A: The table should be updated to include the "header" fields from their response as meta data in the YQL response. A.18. What is the purpose of the produces attribute in the select element? The YQL Guide states that the produces attribute specifies "the type of data coming back from the Web service." Which Web service? YQL? The service the table is getting data from?

3 4

http://backyard.yahoo.com/tools/g/employee/profile?user_id=jcondit http://twiki.corp.yahoo.com/view/YOS/YahooIt 5 http://developer.yahoo.com/yql/console/?q=select%20*%20from%20yahoo.shortener%20where%20url%3D&apos;http%3A%2F%2Fy.ahoo.it%2FATDvNGdZ&apos;&amp;env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys 6 http://developer.yahoo.com/search/boss/boss_guide/ch02s02.html 7 http://developer.yahoo.com/yql/console/#h=select%20title%2Cabstract%20from%20search.web%20where%20query%3D%22pizza%22

Yahoo! Developer Network

40

September 8, 2011

FAQ

For example, I have requested JSON from Picasa in the table definition (alt=json) and specified XML and JSON for this attribute in my table, and it didn't make any difference in the returned response (I'm asking YQL to return JSON). References: XML: http://yql.popmonkey.com/v2/google.picasa.album.photos.xml
8

JSON: http://yql.popmonkey.com/v2/google.picasa.album.photos.json.xml 9 A: Its the Web service you specify in the URL. If you use execute, the produces attribute doesnt really apply to you. If you let YQL just mapkeys directly as a table it makes a big difference. A.19. Where can I find useful sample YQL queries? See the following for example YQL queries: Zach Graves' YQL examples10 YQL Console11 Christian Heilmann's YQL examples12 A.20. Why isn't the console working? If you're using an internal host for the YQL Console, such as http://yosbeta.pipes.yahoo.com/v1/public/yql?q=show%20tables,check to see if you're on VPN. If so, you need to go through the socks proxy to access that host A.21. What should I do if I have a performance issue with YQL? Please open a YQL/General ticket13 with your performance questions, include query and table information as well as any additional relevant performance data points. See YQL Performance14 for performance information and diagrams. A.22. How do you monitor the performance of YQL? YQL uses YMON external and internal and Dashkebab to monitor performance. YWA also collects sample 2% data A.23. What is the purpose of the 'string' type attributes? The type values of the key are at this point just providing a description about the type of data that should be present in the key. YQL does not actually bind based on the type variable, and as you can see from the schema, this attribute is optional, so you may choose to exclude it if you dont see value.

8 9

http://yql.popmonkey.com/v2/google.picasa.album.photos.xml http://yql.popmonkey.com/v2/google.picasa.album.photos.json.xml 10 http://twiki.corp.yahoo.com/view/YOS/YqlExamples 11 http://developer.yahoo.com/yql/console 12 http://spreadsheets.google.com/ccc?key=0AhphLklK1Ve4dFJLZ1pYRmpPcDVTMzBMRloyNHdCTmc&hl=en&authkey=CL6a1ooC#gid=0 13 http://bug.corp.yahoo.com/enter_bug.cgi?product=YQL 14 http://twiki.corp.yahoo.com/view/YOS/YQLPerformance

Yahoo! Developer Network

41

September 8, 2011

FAQ

A.24. Does YQL support WS rate limiting integration? YQL supports rate limiting integration on the set key at the table level. See the Rate Limiter Administration Dashboard15. A.25. Is rate limiting supported for nested queries? Inner queries are processed directly through the engine and are not seen as HTTP inbound traffic. Inbound and REST traffic is what counts towards YDoD; by that token any y.query is considered an internal execute. Note: This is separate from abuse or resource limits, which are internal engine concepts. Also, today any OAuth traffic is monitored but not enacted. A.26. Does YQL support Java in the execute element or Java tables? Java execute16 is not ready for production yet; please provide reasons and use cases for this integration as well as performance issues with JavaScript. A.27. Is there a MySQL connector table? YQL provides a MySQL connector [21], which has some outstanding aspects such as rotation management; please help test and log any issues/enhancements. A.28. What are your plans for support of Java implementation of YQL? Implementation exists, however is not well isolated at this point; planned on 2011 Q2/Q3 roadmap A.29. Where is the YQL roadmap? See the YQL Roadmap17 twiki. A.30. How often does YQL come out with releases? YQL releases about ever 6-8 weeks and follows YOS platform release nomenclature/flow. For more information, see past releases and the features released18.

15 16 17

http://rladmin.yosws.yahoo.com:9999/services.php#page=10 http://twiki.corp.yahoo.com/view/YOS/YqlExecuteInJava http://twiki.corp.yahoo.com/view/YOS/YQL#Roadmap 18 http://twiki.corp.yahoo.com/view/YOS/YQL#YQL_Releases

Yahoo! Developer Network

42

September 8, 2011

You might also like