Download all StackDriver Custom Metrics

In the company I work we use StackDriver as a monitoring tool for almost any instance in AWS (Amazon Web Services) we control.

In one moment we had to do cleanup of all custom metrics that we don’t really using (but actually pay for it) ($0.20 per custom metric / month)

So if you have about 100 000 custom metrics, the price become pretty nice.

I have contacted the StackDriver support for a little assistance about getting the list of all custom metrics we have, but I only get „You can use our API to get a list of custom metrics and it appears that the data is returned in JSON format.“

So I try to use it, but when you do a request to the api, you get only the list of 100 custom metrics and link (meta data) to the next page.

So until I wait for the proper support response, I wrote an ugly script that can download all custom metrics from the api itself. It is not perfect, but works.

#!/bin/bash

#
# Author: Peycho Dimitrov
# URL: http://peycho.com
#
# Requirements:
#
# uname
# wc
# sed
# awk
# grep
# python
# curl
#
################
#
# Usage
#
# Type: ./sd-get-metrics.sh -k YOUR-STACKDRIVER-API-KEY -l /path/to/custom-metrics.log
#
################

# Return codes
RET_CODE_OK=0
RET_CODE_ERROR=1

# Set default start variable for next page
NEXT=""

# StackDriver Api URL
SD_URL="https://api.stackdriver.com/v0.2/custom_metrics/"

# Standard usage function
print_help() {
 echo "$0: Usage"
 echo " [-h] Print this help"
 echo " [--sdkey|-k] (MANDATORY) StackDriver Key"
 echo " [--log|-l] (MANDATORY) Log file location"
 exit $RET_CODE_OK
}

# Check for supported operating system
p_uname="$(whereis uname | cut -d' ' -f2)"
if [ ! -x "$p_uname" ]; then
 echo "$0: No UNAME available in the system"
 exit $RET_CODE_ERROR
fi
OS=$("$p_uname")
if [ "$OS" != "Linux" ]; then
 echo "$0: Unsupported OS!"
 echo "$OS"
 exit $RET_CODE_ERROR
fi

# Check if awk is available in the system
p_awk="$(whereis awk | cut -d' ' -f2)"
if [ ! -x "$p_awk" ]; then
 echo "$0: No AWK available in the system!"
 exit $RET_CODE_ERROR
fi

# Check if grep is available in the system
p_grep=$(whereis grep | cut -d' ' -f2)
if [ ! -x "$p_grep" ]; then
 echo "$0: No GREP available in the system!"
 exit $RET_CODE_ERROR
fi

# Check if wc is available in the system
p_wc=$(whereis wc | cut -d' ' -f2)
if [ ! -x "$p_wc" ]; then
 echo "$0: No WC available in the system!"
 exit $RET_CODE_ERROR
fi

# Check if sed is available in the system
p_sed=$(whereis sed | cut -d' ' -f2)
if [ ! -x "$p_sed" ]; then
 echo "$0: No sed available in the system!"
 exit $RET_CODE_ERROR
fi

# Check if curl is available in the system
p_curl=$(whereis curl | cut -d' ' -f2)
if [ ! -x "$p_curl" ]; then
 echo "$0: No curl available in the system!"
 exit $RET_CODE_ERROR
fi

# Parse command line arguments
while test -n "$1"; do
 case "$1" in
 --help|-h)
 print_help
 exit $RET_CODE_ERROR
 ;;
 --sdkey|-k)
 SDKEY=$2
 shift
 ;;
 --log|-l)
 LOG=$2
 shift
 ;;
 *)
 echo "$0: Unknown or Missing Argument: $1"
 print_help
 exit $RET_CODE_ERROR
 ;;
 esac

 shift
done

# Check if mandatory warning level is present
if [ -z "$SDKEY" -o -z "$LOG" ]; then
 echo "$0: Missing mandatory argument!"
 print_help
 exit $RET_CODE_ERROR
fi

# Check LOG file status
if [ ! -f "$LOG" ]; then
 # Try to create the log file
 echo ""$LOG" not found! Trying to create one..."
 touch "$LOG" 2> /dev/null
 if [ $? -eq 0 ]; then
 echo "Successfully created file: "$LOG""
 else
 echo "$0: No read or write permitions to create $LOG"
 exit $RET_CODE_ERROR
 fi
elif [ ! -r "$LOG" ] || [ ! -w "$LOG" ]; then
 echo "$0: No read or write permitions to $LOG"
 exit $RET_CODE_ERROR
else
 echo "Truncate log file and prepare for new data..."
 truncate -s 0 "$LOG" 2> /dev/null
 if [ $? -eq 0 ]; then
 echo "Successfully created file: "$LOG""
 else
 echo "$0: No write permitions to clear "$LOG" data."
 exit $RET_CODE_ERROR
 fi
fi

C="1"

while true
do
 sleep 1
 R_DATA=$($p_curl -s -G -H "x-stackdriver-apikey: "$SDKEY"" "$SD_URL" --data-urlencode "next=$NEXT" | python -m json.tool)
 NEXT=$(echo "$R_DATA" | $p_grep -w '/v0.2/custom_metrics/?next' | $p_awk -F'=' '{print $2}' | $p_sed 's/"//g')
 C_METRICS=$(echo "$R_DATA" | $p_grep -w '"name":' | $p_awk -F'"' '{print $4}' >> "$LOG")

 PAGE=$(echo "$R_DATA" | $p_grep -c '/v0.2/custom_metrics/?next')
 if [ "$PAGE" -ne 1 -o -z "$PAGE" ]; then
 T_METRICS=$(cat "$LOG" | wc -l)
 echo "$(date) Total custom metrics collected: "$T_METRICS""
 echo "$(date) No more pages or NEXT variable is empty... exiting."
 break
 else
 echo "$(date) Next page ("$C") string is $NEXT"
 C=$((C+1))
 continue
 fi
done

Yes I know that is not perfect and there is several other ways to do it, but… it works for me 😉
I hope it will work to you too.

Вашият коментар Отказ