Navigation

Archive | ShellScript

Using cURL, BASH and Google oAuth to access Google Analytics

By on January 11, 2013 in Programming, ShellScript

In this previous post, I used cURL (the command line version) to interact with Google Analytics. I wanted to do the same thing but using oAuth. I took a lot from this page, but there were a few things that I couldn’t get working, and a few things I didn’t know.

Follow Steps 1-6 on this page. These are steps that you need to follow to get your app registered with Google

In step 6, copy down the code, and keep track of it. It needs to be reused every time you need to get a new token. If you loose it, then you need to run step 6 over again. I didn’t know that.

Here is my script. I will jump through the code below it.

#!/bin/bash
CODE="4/v6xr77ewYqhvHSyW6UJ1w7jKwAzu&amp"
CLEINTID="1234567890.apps.googleusercontent.com"
HEADER="Content-Type: application/x-www-form-urlencoded"
CLIENTSECRET="aBcDeFgHiJkLmNoPqRsTuVwXyZ"
REDIRECTURI="urn:ietf:wg:oauth:2.0:oob"

# I keep the ACCESS_TOKEN and the REFRESH_TOKEN in a file.
if [ -s ~/.google ];then
	ACCESS_TOKEN=$(cat ~/.gauth | grep access_token | awk -F"," '{print $2}' | tr -d ' ')
	REFRESH_TOKEN=$(cat ~/.gauth | grep refresh_token | awk -F"," '{print $2}' | tr -d ' ')
else
	# not used before
	NEWTOKEN=$(curl -s -d "code=$CODE&redirect_uri=$REDIRECTURI&client_id=$CLEINTID&scope=&client_secret=$CLIENTSECRET&grant_type=authorization_code" https://accounts.google.com/o/oauth2/token)
	ACCESS_TOKEN=$(echo $NEWTOKEN | awk -F"," '{print $1}' | awk -F":" '{print $2}' | sed s/\"//g | tr -d ' ')
	REFRESH_TOKEN=$(echo $NEWTOKEN | awk -F"," '{print $4}' | awk -F":" '{print $2}' | sed s/\"//g | sed s/}// | tr -d ' ')
	echo access_token , $ACCESS_TOKEN > .google
	echo refresh_token , $REFRESH_TOKEN >> .google
fi
EXPIRED=$(curl -s https://www.googleapis.com/oauth2/v1/tokeninfo?access_token=$ACCESS_TOKEN | grep 'invalid_token')
if [ "$EXPIRED" ]       
then
	echo "EXPIRED"
	REFRESHRETURN=$(curl -s -d "client_secret=$CLIENTSECRET&grant_type=refresh_token&refresh_token=$REFRESH_TOKEN&client_id=$CLEINTID" https://accounts.google.com/o/oauth2/token)
	ACCESS_TOKEN=$(echo $REFRESHRETURN | awk -F"," '{print $1}' | awk -F":" '{print $2}' | sed s/\"//g | tr -d ' ')
	echo access_token , $ACCESS_TOKEN > .gauth
	echo refresh_token , $REFRESH_TOKEN >> .gauth
fi 
AUTH=$ACCESS_TOKEN
# now in your curl code to retrieve the google analytics data, you use --header "Authorization: OAuth $AUTH"

Lines 1-6: I am setting up my variables with data as described in the linked post.
Lines 8-11: I keep track of the current access token and the refresh token in a config file. If the file exists then parse out the values
Lines 12-19: This is the first time this has been run, so I need to create the file, and put in it a new token and the refresh token. Note the refresh token needs to be saved, and is only given to you once. I did not know that.
Line 20: checks to see if the access token is expired.
Lines 21-28: if the access token is expired, use the refresh token to get a new access token and then save it to the file.

That is it. I hope to translate into PowerShell next – I am sure this code exists, but this is how I learn.

Hope this helps someone.

PowerShell script to backup all SharePoint 2010 lists in all webs in all sites

By on October 10, 2012 in PowerShell, ShellScript

More backups the better. I wanted a file level backup of every list. Below I used PowerShell to iterate through all the lists on the server and dump them into a folder

$backupDir="c:\Temp"
foreach ($web in $(Get-SPSite | Get-SPWeb)){
	foreach ($list in $web.Lists) {
	mkdir -force "$backupDir\$($Web.Title.replace(' ',''))\"
	Export-SPWeb $($web.Url) -itemurl "$($list.RootFolder.ServerRelativeUrl)" -path "$backupDir\$($Web.Title.replace(' ',''))\$($list.Title.replace(' ','')).cmp"
	}
}

My git notes

By on September 12, 2012 in Linux, NoteToSelf, ShellScript
Been playing around with git to manage my “Environment”. This is a note to self. May add more at some point
List files that have been “staged”
  • git diff –name-only –cached
To create a “Centro repo”
  • To setup an empty central repo:
    • mkdir /your/path/folder/project.git
    • cd /your/path/folder/project.git
    • git init –bare –shared
  • To add files to the central repo:
    • Go to the existing file structure and setup a new git repo (if it is not there already)
    • cd your/local/workspace/project
    • git init
    • git add .
    • git commit -m “First Commit”
    • git remote add origin server.domain.com:/your/path/folder/project.git
    • git push origin master
  • If you make a change  in your local copy and you want to push it up to the Centro repo
    • git add .
    • git commit -m “This is what changed”
    • git push origin master
  • And to get those changes to other machines
    • git pull origin master
  • And to setup a new machine
    • git clone server.domain.com:/your/path/folder/project.git

Got most of the info from here

BASH, cURL script to retrieve Google Analytics data

By on June 4, 2012 in ShellScript

Last week I wrote this PowerShell code to retrieve Google Analytics data. Below is similar code in BASH using the cURL command.

#!/bin/bash
stty -echo
read -p "Password: " password; echo
stty echo
RESULT=$(curl -s https://www.google.com/accounts/ClientLogin \
--data-urlencode [email protected] --data-urlencode Passwd=$password \
-d accountType=GOOGLE \
-d source=YourSource \
-d service=analytics)


AUTH=$(echo "$RESULT" | grep 'Auth=' | sed  s/Auth=//)

curl -s "https://www.google.com/analytics/feeds/data?ids=ga%3AXXXXXXXX&metrics=ga%3Avisits&start-date=2012-05-01&end-date=2012-05-31&max-results=50" \
--request GET --header "Authorization: GoogleLogin auth=$AUTH"

#Done

Update DynDNS from the command line

By on April 24, 2012 in Linux, Mac, ShellScript

My router is not updating DynDNS correctly, so I wanted to use a simple cron job to do the same. Here is a simple BASH/Shell script to update DynDNS:

IP=$(curl -s icanhazip.com)
curl -v -k -u username:password “https://members.dyndns.org/nic/update?hostname=DNSHOSTNAME.dyndns.org&myip=$IP”

Using cURL to pull Google Reader starred items – Part 2 UnStaring in Google Reader

By on February 28, 2012 in Mac, ShellScript

In the first part of this tutorial, I pulled a couple of variables from the XML feed of my Google Reader’s “starred items”. Now I wanted to “process” the item and UnStar it. This was not easy for me to put together – it was my first attempt at working with the Google API.

First I needed to get authenticated against Google Services for accessing Google Reader. Here is the shell script to do this:

stty -echo
read -p "Password: " password; echo
stty echo
RESULT=$(curl -s https://www.google.com/accounts/ClientLogin \
--data-urlencode [email protected] --data-urlencode Passwd=$password \
-d accountType=GOOGLE \
-d source=MyAppName \
-d service=reader)

This returned some html that included the AUTH code that I needed to add to the header of each Google Reader Request. I used this AUTH to get a Token (my understanding is that if I wanted to edit an item, I needed the token too). Here are the two pieces of code to parse the AUTH get the Token:

AUTH=$(echo "$RESULT" | grep 'Auth=' | sed  s/Auth=//)
TOKEN=$(curl -s --header "Authorization: GoogleLogin auth=$AUTH" http://www.google.com/reader/api/0/token)

Putting the AUTH together with the Token, and the Source and id from this post, you end up with a cURL command that can mark an item as UnStared:

curl -s http://www.google.com/reader/api/0/edit-tag?client=MyAppName --request POST --header "Authorization: GoogleLogin auth=$AUTH" \
--data-urlencode r=user/-/state/com.google/starred  \
--data-urlencode async=true \
--data-urlencode s=$SOURCE \
--data-urlencode i=$OBJID \
--data-urlencode T=$TOKEN

I was surprised when i got this to work. I am still scared of APIs/REST.

Using cURL to pull Google Reader starred items – Part 1 xpath

By on February 28, 2012 in Mac, Programming, ShellScript

A while ago, I wrote a post about using ruby to parse the xml feed of shared starred items in Google Reader. One thing that I did not like about this solution was that I could not get the URL AND mark the item as un-starred. Since I had been playing with REST in these two prior posts, I figured I could re-write my code to pull down starred item’s URLs, and mark them as un-starred. I wanted to eliminate ruby, as I just don’t use it that often and I feel like I am re-inventing the wheel every time. This too a lot longer than I thought it would, but I figured it out (I think)

As this article explains, you can share out your Google reader starred items. First step was the find the needed values from this XML feed and put the values into variables. I turned to xpath (xpath is installed by default on OS X, it is part of the Perl library XML::XPath).

For the second part of this tutorial, where we mark the item as “un-starred”, we need 2 variables form the Xml feed: id & source. My end goal was to put these URLs into Together.app, so I needed the URL too. Title was just for fun.

Here is the non ruby code to pull the variables that I needed out of the XML of the shared starred items in Google Reader.

XML=$(curl -s http://www.google.com/reader/public/atom/user/YOURUSERID/state/com.google/starred?n=1 | xmllint --format -)
OBJID=$(echo "$XML" | xpath "//entry/id" 2>/dev/null | awk -F"[<>]" '{print $3}')
TITLE=$(echo "$XML" | xpath "//entry/title" 2>/dev/null | awk -F"[<>]" '{print $3}')
URL=$(echo "$XML" | xpath "//entry/link/@href" 2>/dev/null | sed 's/\"//g' | sed 's/href\=//g' | sed 's/\ //g')
SOURCE=$(echo "$XML" | xpath "//entry/source/@gr:stream-id" 2>/dev/null | sed "s/gr:stream-id=//g" | sed "s/\"//g" | sed 's/\ //g')

After getting those variables, I wanted to put the URL in together.app (this has not changed since the previous post). Here is that code:

echo "Adding $TITLE to Together"
osascript << EOT
tell application "Together" to import url "$URL" as web PDF
EOT

OS X: Running a script when a USB drive is inserted

By on July 15, 2011 in Mac, ShellScript

I rsync all my data to a USB drive that I keep at work. I wanted a way to have my rsync script automatically run when I plugged in the drive – kinda like Time Machine.

It ended up being pretty simple. All I needed to do is create and AppleScript and attach it to a “Folder Action” for the /Volumes folder. This script below is launched when a new item is added to the /Volumes folder, i.e. when you insert a new volume. This script will try to run a BASH script if it exists on that volume (.OnInsert)

on adding folder items to this_folder after receiving these_items
	repeat with current_item in these_items
		try
			do shell script POSIX path of current_item & ".OnInsert"
		end try
	end repeat
end adding folder items to

Save this as a .scpt file and put it in ~/Library/Workflows/Applications/Folder\ Actions folder.
Next, right click the /Volumes folder and select Services -> Folder Action Setup and attach the script you just created
AutoMagic!