Using Google Cloud Storage - Part 1

This is my first attempt at a tutorial for Google Cloud Storage.

Using Google Cloud Storage… it's a service that let's you read/write data objects (a blob of data of what ever type you want).  Right now, you get 5GB of storage for free and you can transfer 20 GB per month. Beyond that, you'll have to check the pricing.

This is pretty much what is on https://developers.google.com/storage/docs/getting-started and https://developers.google.com/storage/docs/hellogooglestorage I've tweaked it a bit to fit my style. I'm doing this on a mac with fink and splitting between two acounts. I want to end up with a way to transfer files from one group to another. I started off by watching this 2011 Google IO video:

Google I/O 2011: Storing Your Application's Data in the Google Cloud

First setting up a virtual env in python to isolate us from messing up any existing python installs.

cd ~/Desktop
mkdir try-gs
cd try-gs
fink install virtualenv-py27
virtualenv ve
source ve/bin/activate
wget http://commondatastorage.googleapis.com/pub/gsutil.tar.gz
tar xf gsutil.tar.gz
pushd gsutil

# gsutil does not have a proper setup.py that will work with virtualenv
# so force it to play along
perl -pi -e "s|/usr|`pwd`/../ve|g" setup.py

python setup.py install
popd

You should now have a working virtualenv with gsutil installed in it. Your normal python will not be effective. You will have to rerun the <b>activate</b> from any other shell that you want to have access to gsutil. The virtualenv named "ve" will change your prompt:

$ pwd
/Users/schwehr/Desktop/try-gs
(ve)
$ 

Let's test out setup.

type python
# python is hashed (/Users/schwehr/Desktop/try-gs/ve/bin/python)
type gsutil
# gsutil is /Users/schwehr/Desktop/try-gs/ve/bin/gsutil
gsutil --help 
# lots of help scrolls by

# Try a world readable test data set
gsutil ls gs://uspto-pair/applications/0800401*
# gs://uspto-pair/applications/08004010.zip
# gs://uspto-pair/applications/08004011.zip
# gs://uspto-pair/applications/08004012.zip
# gs://uspto-pair/applications/08004013.zip
# gs://uspto-pair/applications/08004016.zip
# gs://uspto-pair/applications/08004017.zip
# gs://uspto-pair/applications/08004019.zip
gsutil ls -l gs://uspto-pair/applications/08004010.zip 
#       3410  2010-09-29T20:14:15  gs://uspto-pair/applications/08004010.zip
# TOTAL: 1 objects, 3410 bytes (3.33 KB)
gsutil cp gs://uspto-pair/applications/08004010.zip .
md5sum 08004010.zip 
# 04b4877377c16b1b49f19b0c7e161d04  08004010.zip

I have yet to write up using the oauth2 authentication.  Hopefully, I'll get that done soon.

See Also:

• http://code.google.com/p/gsutil/
• Issue 92 covers using pypi / pip / distribute / setuptools