Warning: This page is about Google's older APIs, the Google Data APIs; it's relevant only to the APIs that are listed in the Google Data APIs directory, many of which have been replaced with newer APIs. For information about a specific new API, see the new API's documentation. For information about authorizing requests with a newer API, see Google Accounts Authentication and Authorization.
Ryan Boyd, Google Data APIs TeamSeptember 2007
- Introduction
- Obtaining and installing cURL
- Authenticating to a Google Data Service
- Retrieving feeds and entries
- Updating entries
- Creating entries
- Deleting entries
- Uploading media objects
- Other command-line tools
- Conclusion
Introduction
At heart, Google Data APIs use Atom feeds and entries (XML) as a data format and HTTP as a protocol for data transmission - extending the Atom Publishing Protocol. We publish a number of client libraries to make interacting with Google Data APIs easier. However, you're always welcome to use lower-level tools to work with our services, and it's pretty easy to do with a little guidance.
cURL is a command-line application for performing requests using a variety of protocols including HTTP. cURL is often used by developers to test Google Data services, as it supports the HTTP functionality required to interact with the APIs at a low level.
cURL only provides support for performing the HTTP communication, so knowledge of the Google Data protocol, the service-specific protocol and the XML data format used is a prerequesite for working with the application. Some other tools are mentioned in this article for making these tasks easier.
This article uses examples based upon the Picasa Web Albums data API. However, all these examples can be readily applied to other Google Data APIs.
Obtaining and installing cURL
cURL is commonly available on a default install of many UNIX/Linux platforms. Try typing curl
in your favorite shell to see if the tool is installed and is in your PATH
. If you don't have the tool installed, visit the download page on the cURL website to obtain the official source or a user-contributed binary package. Note that the command-line tool uses the libcurl
library, which may be offered as a separate download package, so, if you're not compiling from the source, be sure to download a 'binary' package instead of a 'libcurl' package. The SSL-enabled packages are required if you wish to use cURL to acquire authentication tokens or to access some Google Data services which require using SSL for requests.
Authenticating to a Google Data Service
Authenticated Google Data requests are performed by adding an HTTP header to the request which contains either a ClientLogin (desktop/mobile apps) or AuthSub (web apps) authentication token. For the purposes of testing using cURL, ClientLogin is the easier method and is documented below. AuthSub authentication headers could be used with cURL, but the more-advanced process of obtaining the tokens is out of scope for this article.
Using ClientLogin
ClientLogin is intended for installed (desktop/mobile) applications. With this method of authentication, the application using Google Data APIs directly handles the username and password of the user.
An authentication request for ClientLogin takes a username, password, and service name as form post variables. These variables are passed as the Email
, Passwd
, and service
arguments, respectively. This request yields a response with several tokens, one of which can be used to make requests to the Google Data service. Note that data arguments passed with curl
must be URL-encoded if they contain non-ASCII characters, which often appear in Email
and Passwd
arguments. You can ask curl
to URL-encode these arguments by using the --data-urlencode
flag.
Example Request:
curl https://www.google.com/accounts/ClientLogin \ --data-urlencode Email=brad.gushue@example.com --data-urlencode Passwd=new+foundland \ -d accountType=GOOGLE \ -d source=Google-cURL-Example \ -d service=lh2
Example Response:
SID=DQAAAHYBADCv2pSv7nflacDNwz3zEDUGtrSvNVDcpkSfddi77b3U5sEaHmP8YLWhmA36F9rk85mL8J5dqo4apn0T1vKz0fPGI9Xtnuet6cuE2ZzYvrNIwbSC_HjTqF4zudNQnnlDuD2wqZT-g1qXI8KhGAQZV4NexHZoQPlabTsGuRZeIBxj1A LSID=EUBBBIaBADCl-kNxvRVmcQghpt3cqSMfEooKR9flLOUZqwgP9OrZS83gse-KSdTNeXhxsET7FYenDhceP9lIPOmesH-t9qh-AWUHjjMdZEbUNeF9mWyzln6Z-FajaiG-cVFkqW0ZJ8ZbnCP30xXj6xFK6QxaAcqy_9Pej8jhEnxS9E61ftQGPg Auth=EUBBIacAAADK-kNxvRVmcQghpt3cqSMfEooLNMflLNIQqwgP9OrZS83gs-KSdTNeXhxsET7FYePWmaD8Vsy1V4LSUGMUP48Je2TO8OcjBj6HgAtPhiZeX-gKDfagZDK44j4n-Tkb44nhOnp2_QPSnBj3Z2vYwOEDjjG3Q53aQVC2132JKOuGh
Please see the ClientLogin documentation for specific information on the parameters used in the above request. In this example, the service we're using is the Picasa Web Albums data API. The service name (service
) is lh2
. The service names for other Google Data services can be found in the Google Data APIs FAQ page.
The value of the Auth
token in the above response is the only value needed for authentication to Google Data services. The value of this token is formed into an HTTP header which is then used for each request to a Google Data service.
curl --silent --header "Authorization: GoogleLogin auth=EUBBIacAAADK\ -kNxvRVmcQghpt3cqSMfEooLNMflLNIQqwgP9OrZS83gs-KSdTNeXhxs\ ET7FYePWmaD8Vsy1V4LSUGMUP48Je2TO8OcjBj6HgAtPhiZeX-gKDfag\ ZDK44j4n-Tkb44nhOnp2_QPSnBj3Z2vYwOEDjjG3Q53aQVC2132JKOuGh" \ "http://picasaweb.google.com/data/feed/api/user/default"
Note: The method of escaping newline characters with backslash characters ('\') above does not work in the Windows command shell, so you must enter the entire command on one line if you're running curl
on Windows.
Retrieving feeds and entries
In Google Data APIs, retrieving feeds and entries is done by performing an HTTP GET
on a URL, with an optional set of query parameters. Because we are performing a GET
request, only the auth header and the URL are required to be passed to curl
. The example below will continue using the Picasa Web Albums data API and is used to retrieve a list of albums owned by the authenticated user. Note that we have shortened the auth token to ABCDEFG
in this example, but the full token (e.g. EUBBIacA
...32JKOuGh
from above) should be used in its place.
curl --silent --header "Authorization: GoogleLogin auth=ABCDEFG" "http://picasaweb.google.com/data/feed/api/user/default"
This will return an unformatted blob of XML:
<?xml version='1.0' encoding='UTF-8'?><feed xmlns='http://www.w3.org/2005/Atom' xmlns:openSearch='http://a9.com/-/spec/opensearchrss/1.0/' xmlns:geo='http://www.w3.org/2003/01/geo/wgs84_pos#' xmlns:gml='http://www.opengis.net/gml' xmlns:georss='http://www.georss.org/georss' xmlns:photo='http://www.pheed.com/pheed/' xmlns:media='http://search.yahoo.com/mrss/' xmlns:batch='http://schemas.google.com/gdata/batch' xmlns:gphoto='http://schemas.google.com/photos/2007'><id>http://picasaweb.google.com/data/feed/base/user/brad.gushue</id><updated>2007-09-13T21:30:21.454Z</updated>...</entry></feed>
There are some decent tools for formatting this output to make it more human-readable, including tidy. The easiest way to use tidy is to pipe the output from the curl command to tidy like the following:
curl --silent --header "Authorization: GoogleLogin auth=ABCDEFG" "http://picasaweb.google.com/data/feed/api/user/default" | tidy -xml -indent -quiet
This results in a much more readable feed, like the following:
<?xml version='1.0' encoding='utf-8'?> <feed xmlns='http://www.w3.org/2005/Atom' xmlns:openSearch='http://a9.com/-/spec/opensearchrss/1.0/' xmlns:geo='http://www.w3.org/2003/01/geo/wgs84_pos#' xmlns:gml='http://www.opengis.net/gml' xmlns:georss='http://www.georss.org/georss' xmlns:photo='http://www.pheed.com/pheed/' xmlns:media='http://search.yahoo.com/mrss/' xmlns:batch='http://schemas.google.com/gdata/batch' xmlns:gphoto='http://schemas.google.com/photos/2007'> <id>http://picasaweb.google.com/data/feed/api/user/brad.gushue</id> <updated>2007-09-13T21:47:07.337Z</updated> <category scheme='http://schemas.google.com/g/2005#kind' term='http://schemas.google.com/photos/2007#user' /> <title type='text'>brad.gushue</title> <subtitle type='text'></subtitle> <icon> http://lh6.google.com/brad.gushue/AAAAj9zigp4/AAAAAAAAAAA/RiMAlXV4MFI/s64-c/brad.gushue</icon> <link rel='http://schemas.google.com/g/2005#feed' type='application/atom+xml' href='http://picasaweb.google.com/data/feed/api/user/brad.gushue' /> <link rel='alternate' type='text/html' href='http://picasaweb.google.com/brad.gushue' /> <link rel='self' type='application/atom+xml' href='http://picasaweb.google.com/data/feed/api/user/brad.gushue?start-index=1&max-results=1000' /> <author> <name>Brad</name> <uri>http://picasaweb.google.com/brad.gushue</uri> </author> <generator version='1.00' uri='http://picasaweb.google.com/'> Picasaweb</generator> <openSearch:totalResults>8</openSearch:totalResults> <openSearch:startIndex>1</openSearch:startIndex> <openSearch:itemsPerPage>1000</openSearch:itemsPerPage> <gphoto:user>brad.gushue</gphoto:user> <gphoto:nickname>Brad</gphoto:nickname> <gphoto:thumbnail> http://lh6.google.com/brad.gushue/AAAAj9zigp4/AAAAAAAAAAA/RiMAlXV4MFI/s64-c/brad.gushue</gphoto:thumbnail> <entry> <id> http://picasaweb.google.com/data/entry/api/user/brad.gushue/albumid/9810315389720904593</id> <published>2007-05-23T04:55:52.000Z</published> <updated>2007-05-23T04:55:52.000Z</updated> <category scheme='http://schemas.google.com/g/2005#kind' term='http://schemas.google.com/photos/2007#album' /> <title type='text'>Trip To Italy</title> <summary type='text'>This was the recent trip I took to Italy.</summary> <rights type='text'>public</rights> <link rel='http://schemas.google.com/g/2005#feed' type='application/atom+xml' href='http://picasaweb.google.com/data/feed/api/user/brad.gushue/albumid/9810315389720904593' /> <link rel='alternate' type='text/html' href='http://picasaweb.google.com/brad.gushue/TripToItalyV2' /> <link rel='self' type='application/atom+xml' href='http://picasaweb.google.com/data/entry/api/user/brad.gushue/albumid/9810315389720904593' /> <link rel='edit' type='application/atom+xml' href='http://picasaweb.google.com/data/entry/api/user/brad.gushue/albumid/9810315389720904593/123456' /> <author> <name>Brad</name> <uri>http://picasaweb.google.com/brad.gushue</uri> </author> <gphoto:id>9810315389720904593</gphoto:id> <media:group> ... </media:group> </entry> <entry> ... </entry> </feed>
Individual entries can be retrieved in the same way by providing the URL for the entry, as opposed to a feed URL.
Updating entries
Entries in Google Data APIs are updated by doing an HTTP PUT
to the edit URL with a new copy of the entry's XML in the body of the request.
- Retrieve the entry using the
atom:link/@rel='self'
URL value - Update the entry locally to make the needed changes
PUT
the entry back to the server, using theatom:link/@rel='edit'
URL value
1. Retrieving the entry
The entry can be retrieved using one of the two URLs bolded in the feed block above. The URL needed is the href
value for the link
element with a rel='self'
.
curl --silent --header "Authorization: GoogleLogin auth=ABCDEFG" "http://picasaweb.google.com/data/entry/api/user/brad.gushue/albumid/9810315389720904593"
2. Updating the entry locally
After you retrieve the entry, the entry needs to be updated using a local text editor or application to make any needed changes to the entry. In the command above to retrieve an entry, we did not pipe the results to tidy
as we have done in the previous examples. The result is XML that represents the same data, but has different formatting than the version piped to tidy
. For the purposes of hand editing an entry, using tidy
can often make working with the XML easier.
Note: Please remember to include all XML namespace definitions which are used as attributes to the atom:entry
when you post your new entry. Omitting these will cause parsing exceptions. Also, tidy
will replace the spaces between the namespace definitions with newline characters. While this is valid XML, Google Data services do not accept it at this time. If you are using tidy
, please be sure to add additional spaces between these attributes on the entry
element.
3. Updating the entry on the server
Using the edit
URL, you need to PUT
a copy of the entry to the service using cURL. A header to indicate the type of content being sent to the server needs to be added. The following snippet assumes that the file with the updated entry is saved in updated_entry.xml.
curl --silent --data-binary "@updated_entry.xml" --request PUT --header "Content-Type: application/atom+xml" --header "Authorization: GoogleLogin auth=ABCDEFG" "http://picasaweb.google.com/data/entry/api/user/brad.gushue/albumid/9810315389720904593/123456"
Creating entries
Entries in Google Data APIs are created by doing an HTTP POST
to the post URL with a new entry. The atom:id
is assigned by the server, and thus not necessary to include in new entries. The easiest way to create a new entry is to take an old entry and modify it. The following example will do just that.
- Retrieve a template entry using the
atom:link/@rel='self'
- Modify the template entry locally to remove unnecessary information and make the needed changes
POST
the entry back to the server, using thepost
URL for the feed. This is either found in the retrieved feed as thehref
value for thelink
element with arel='http://schemas.google.com/g/2005#post'
, or in the documentation for the service on http://code.google.com.
1. Retrieve a template entry
A single entry can be retrieved using the href
value of a link
element with a rel='self'
in the same way as an entry was retrieved before updating it in the above example.
curl --silent --header "Authorization: GoogleLogin auth=ABCDEFG" "http://picasaweb.google.com/data/entry/api/user/brad.gushue/albumid/9810315389720904593" | tidy -xml -indent -quiet > template_entry.xml
The response, after using tidy
will look something like:
<entry xmlns='http://www.w3.org/2005/Atom' xmlns:exif='http://schemas.google.com/photos/exif/2007' xmlns:geo='http://www.w3.org/2003/01/geo/wgs84_pos#' xmlns:gml='http://www.opengis.net/gml' xmlns:georss='http://www.georss.org/georss' xmlns:photo='http://www.pheed.com/pheed/' xmlns:media='http://search.yahoo.com/mrss/' xmlns:batch='http://schemas.google.com/gdata/batch' xmlns:gphoto='http://schemas.google.com/photos/2007'> <id> http://picasaweb.google.com/data/entry/api/user/brad.gushue/albumid/9810315389741123981</id> <published>2007-05-23T04:55:52.000Z</published> <updated>2007-05-23T04:55:52.000Z</updated> <category scheme='http://schemas.google.com/g/2005#kind' term='http://schemas.google.com/photos/2007#album' /> <title type='text'>Trip To Italy</title> <summary type='text'>This was the recent trip I took to Italy.</summary> <rights type='text'>public</rights> <link rel='http://schemas.google.com/g/2005#feed' type='application/atom+xml' href='http://picasaweb.google.com/data/feed/api/user/brad.gushue/albumid/9810315389741123981' /> <link rel='alternate' type='text/html' href='http://picasaweb.google.com/brad.gushue/TripToItaly' /> <link rel='self' type='application/atom+xml' href='http://picasaweb.google.com/data/entry/api/user/brad.gushue/albumid/9810315389741123981' /> <link rel='edit' type='application/atom+xml' href='http://picasaweb.google.com/data/entry/api/user/brad.gushue/albumid/9810315389741123981/1179896152905401' /> <author> <name>Brad</name> <uri>http://picasaweb.google.com/brad.gushue</uri> </author> <gphoto:id>9810315389741123981</gphoto:id> <gphoto:name>TripToItaly</gphoto:name> <gphoto:location></gphoto:location> <gphoto:access>public</gphoto:access> <gphoto:timestamp>1179896152000</gphoto:timestamp> <gphoto:numphotos>0</gphoto:numphotos> <gphoto:numphotosremaining>500</gphoto:numphotosremaining> <gphoto:bytesUsed>0</gphoto:bytesUsed> <gphoto:user>brad.gushue</gphoto:user> <gphoto:nickname>Brad</gphoto:nickname> <gphoto:commentingEnabled>true</gphoto:commentingEnabled> <gphoto:commentCount>0</gphoto:commentCount> <media:group> <media:title type='plain'>Trip To Italy</media:title> <media:description type='plain'>This was the recent trip I took to Italy.</media:description> <media:keywords></media:keywords> <media:content url='http://lh5.google.com/brad.gushue/ADFUFKqeA5E/AAAAAAAAABc/V6Sga_Z03WU/TripToItaly.jpg' type='image/jpeg' medium='image' /> <media:thumbnail url='http://lh5.google.com/brad.gushue/ADFUFKqeA5E/AAAAAAAAABc/V6Sga_Z03WU/s160-c/TripToItaly.jpg' height='160' width='160' /> <media:credit>Brad</media:credit> </media:group> </entry>
2. Modify the template entry
We want to create an album called "Curling in Canada" with pictures from our recent curling match. Google Data allows you to drop Atom elements that the server supplies values for, so to create this simple template entry, we'll remove the atom:id
, atom:published
, atom:updated
, atom:author
, and the various atom:link
elements in the feed. This will give us a stripped-down template entry. The entry then needs to be modified to represent the new album we are creating:
<entry xmlns='http://www.w3.org/2005/Atom' xmlns:gphoto='http://schemas.google.com/photos/2007'> <category scheme='http://schemas.google.com/g/2005#kind' term='http://schemas.google.com/photos/2007#album' /> <title type='text'>Curling in Canada</title> <summary type='text'>Pictures of all my curling matches in Canada.</summary> <gphoto:location>Canada</gphoto:location> <gphoto:access>public</gphoto:access> <gphoto:commentingEnabled>true</gphoto:commentingEnabled> </entry>
Note: Please remember to include all XML namespace definitions which are used as attributes to the atom:entry
when you post your new entry. Omitting these will cause parsing exceptions. Also, tidy
will replace the spaces between the namespace definitions and replace them with newline characters. While this is valid XML, Google Data services do not accept it at this time. If you are using tidy
, please be sure to add additional spaces between these attributes on the entry
element.
3. Posting the new entry to the server
The curl
command for posting a new entry to the server is very similar to updating an existing entry except that the URL is different:
curl --silent --request POST --data-binary "@template_entry.xml" --header "Content-Type: application/atom+xml" --header "Authorization: GoogleLogin auth=ABCDEFG" "http://picasaweb.google.com/data/feed/api/user/brad.gushue" | tidy -xml -indent -quiet
If the post was successful, the resulting XML output is a copy of the newly-created entry. This entry will include things which the server generated at the time the entry was created, including the values for the atom:id
, atom:published
, atom:updated
, and atom:link
elements. The resulting link
values can be used to edit or delete the entry, provided no additional changes are made in the interim.
Deleting entries
Deleting entries is very similar to updating entries, except an HTTP DELETE
method is used instead of an HTTP PUT
and no data is required to be sent. Also like the update request, the edit
URL is used as the target of the HTTP request.
curl --silent --request DELETE --header "Authorization: GoogleLogin auth=ABCDEFG" "http://picasaweb.google.com/data/entry/api/user/brad.gushue/albumid/9810315389720904593/123456"
Uploading media objects
An important feature of the Picasa Web Albums data API and the Documents List data API is the ability to upload binary objects. cURL can easily accomplish uploading binary data and a slug header. However, the Documents List data API currently requires posting the XML along with the binary data as a MIME multipart message. Forming the multipart message is out of scope for this article.
The example below shows how to upload a picture called sweeping_the_rock.png
to a Picasa Web Album with the title "Sweeping the rock":
curl --silent --request POST --data-binary "@sweeping_the_rock.png" --header "Slug: Sweeping the rock" --header "Content-Type: image/png" --header "Authorization: GoogleLogin auth=ABCDEFG" "http://picasaweb.google.com/data/feed/api/user/brad.gushue/albumid/5113621341847124417" | tidy -xml -indent -quiet
Other command-line tools
Some developers prefer learning or debugging using other command-line tools.
Popular tools include:
- telnet, openssl are used for making raw socket connections (plain text and ssl-based, respectively) to web servers and can be used to interact with Google Data services. Note that not all Google Data services may support SSL. Here's how you open up the connections:
telnet picasaweb.google.com 80
(Picasa Web Albums data API)openssl s_client -connect www.google.com:443
(Google Calendar data API and other services on www.google.com)
POST /data/feed/api/user/brad.gushue HTTP/1.1 Host: picasaweb.google.com Authorization: GoogleLogin auth=ABCDEFG Content-Length: 493 <entry xmlns='http://www.w3.org/2005/Atom' xmlns:gphoto='http://schemas.google.com/photos/2007'> <category scheme='http://schemas.google.com/g/2005#kind' term='http://schemas.google.com/photos/2007#album' /> <title type='text'>Curling in Canada</title> <summary type='text'>Pictures of all my curling matches in Canada.</summary> <gphoto:location>Canada</gphoto:location> <gphoto:access>public</gphoto:access> <gphoto:commentingEnabled>true</gphoto:commentingEnabled> </entry>
When sending raw HTTP data, be aware that thePOST
andPUT
operations will require computing the value for aContent-Length
header. You can use the UNIX toolwc
to compute this value. Place all the content of the HTTP body into a text file such astemplate_entry.xml
(example used above) and runwc -c template_entry.xml
. It is often difficult to debug if you accidentally use an incorrect value for theContent-Length
header. - wget is typically used to download data from a web server to a local file. However,
wget
has lots of options which make it capable of performing all the different types of requests needed to interact with Google Data services.. Here's an example of how to usewget
toPOST
a new album entry to Picasa Web Albums:wget --post-file template_entry.xml --header "Authorization: GoogleLogin auth=ABCDEFG" --header "Content-Type: application/atom+xml" "http://picasaweb.google.com/data/feed/api/user/brad.gushue"
- xsltproc is a tool to apply XSL transformations (XSLT) to XML documents. It can be used to easily extract desired bits of data from an XML entry or feed returned by Google Data APIs, or to generate new or updated entries.
Conclusion
As you've seen, cURL and several other command-line tools can be used to easily interact with Google Data services using raw XML and HTTP. Please join us in the API specific forums if you have any questions about using these tools with your favorite Google Data API.