Monday, December 6, 2010

[TECH] amazon route 53

Amazon Route 53 provides cheap scalable DNS. unfortunately, it has no GUI and the XML interface is too verbose to use by hand. fortunately, it's easy to write a script for DNS updates, and it's even easier to grab an existing script (see below -- some of it is truncated, but it seems to cut & paste correctly). the script expects AWS-provided 'dnscurl.pl' to be available in the current directory, and '.aws-secrets' to contain working credentials. to set these up, follow AWS-provided instructions here. you do not need to "create a hosted zone"; the script below creates zones as necessary. the script expects a 'zones' subdirectory. each zone (remember, a "zone" corresponds to a domain-name) should have a subdirectory in 'zones.' each record type should have a subdirectory named A, MX, or CNAME (according to the record type) in the zone subdirectory. each record type directory should have arbitrarily-named files containing the record data. an example should clarify the format better; here are the files involved in one zone:
$ find zones -type f | egrep zerosum42.com
zones/zerosum42.com/rrs
zones/zerosum42.com/CNAME/1
zones/zerosum42.com/CNAME/2
zones/zerosum42.com/id
zones/zerosum42.com/A/1
zones/zerosum42.com/A/2
zones/zerosum42.com/MX/1
the id and rrs files are automatically maintained (id contains a zone identifier intelligible to AWS and rrs contains the records confirmed by AWS last time the script was run). the remainder of the files contain DNS data in a straightforward format (though non-standard):
$ echo zones/*/*/* | tr ' ' '\n' | egrep zerosum42.com | ( while read NAME; do echo "====================> $NAME"; cat "$NAME"; done )
====================> zones/zerosum42.com/A/1
zerosum42.com 600
204.236.154.100
====================> zones/zerosum42.com/A/2
myhomepc.zerosum42.com 1
127.0.0.1
====================> zones/zerosum42.com/CNAME/1
www.zerosum42.com 600
zerosum42.com
====================> zones/zerosum42.com/CNAME/2
testing.zerosum42.com 600
zerosum42.com
====================> zones/zerosum42.com/MX/1
zerosum42.com 600
aspmx.l.google.com            10
alt1.aspmx.l.google.com       20
alt2.aspmx.l.google.com       20
aspmx2.googlemail.com         30
aspmx3.googlemail.com         30
aspmx4.googlemail.com         30
aspmx5.googlemail.com         30
basically, the first line contains a DNS "key" and TTL (in seconds) while subsequent lines contain DNS "values" one-per-line. note that the script is very sensitive to additional blank lines in the input; be careful. miscellaneous notes:
  • TTL of 600 seconds in 10 minutes.
  • trailing dot: example.com. vs example.com -- the trailing dot is usually required by DNS but it is supplied by the script; don't put it in yourself or there will be two dots and things get very confused. the script knows about record types and basically -always- supplies the dot.
  • only records currently supported are A, MX and CNAME. to get more, edit the script.
that's pretty much it. to invoke the script, pass the names of the zones to be updated as separate parameters. to update all zones, use the following shortcut (assuming your shell is bash or similar):
$ ls zones | xargs ./push.sh
here's the actual script. share and enjoy. oh yeah, forgot to mention software prerequisites ... you need the following ubuntu packages: curl, xmlstarlet. xmlstarlet is just used for final formatting for display to the user; you can probably chop those parts out of the script without too much trouble if you don't want xmlstarlet for some reason. curl is a dependency of the AWS-provided 'dnscurl.pl' that actually interfaces with AWS, so harder to fix that part if for some reason you can't get curl.
#!/bin/bash
# push.sh
# copyright (c) 2010 by andrei borac (zerosum42 AT gmail DOT com)
# this code is hereby PUBLIC DOMAIN, but there is NO WARRANTY

set -o errexit
set -o nounset
set -o pipefail

TMPINP=/tmp/aws-route53-inp-$$
TMPOUT=/tmp/aws-route53-out-$$
TMPRRS=/tmp/aws-route53-rrs-$$

function dnscurl_get()
{
  echo "dnscurl_get('$1')"
  ./dnscurl.pl --keyname my-aws-account -- -H "Content-Type: text/xml; charset=UTF-8" https://route53.amazonaws.com/2010-10-01/"$1" > "$TMPOUT"
}

function dnscurl_post()
{
  echo "dnscurl_post('$1')"
  ./dnscurl.pl --keyname my-aws-account -- -H "Content-Type: text/xml; charset=UTF-8" -X POST --upload-file "$TMPINP" https://route53.amazonaws.com/2010-10-01/"$1" > "$TMPOUT"
}

function dnscurl_delete()
{
  echo "dnscurl_delete('$1')"
  ./dnscurl.pl --keyname my-aws-account -- -H "Content-Type: text/xml; charset=UTF-8" -X DELETE https://route53.amazonaws.com/2010-10-01/"$1" > "$TMPOUT"
}

###
# obtains current record sets
###
function dnscurl_obtain_rrs()
{
  dnscurl_get 'hostedzone/'"$1"'/rrset?maxitems=100'
  ### ENABLE BELOW FOR DEBUGGING:
  #echo "<========== ENTER EXISTING RECORD SETS (RAW)"
  #cat "$TMPOUT"
  #echo
  #echo "<========== LEAVE EXISTING RECORD SETS (RAW)"
  cat "$TMPOUT" | sed -e 's!<ResourceRecordSet>!'"\n"'<ResourceRecordSet>!g' | sed -e 's!</ResourceRecordSet>.*!</ResourceRecordSet>!' | ( egrep '^<ResourceRecordSet>' || true ) | ( egrep -v '<Type>(NS|SOA)</Type>' || true ) | cat > "$TMPRRS"
  RRSZ="`stat -c %s "$TMPRRS"`"
  if (($RRSZ<8))
  then
    rm "$TMPRRS"
  fi
  echo "<========== ENTER ROUTE53 CURRENT RECORD SETS"
  if [ -f "$TMPRRS" ]
  then
    ( echo '<SET>'; cat "$TMPRRS"; echo '</SET>' ) | xmlstarlet fo -o | ( egrep -v 'ResourceRecord(|s|Set)>' || true )
  fi
  echo "<========== LEAVE ROUTE53 CURRENT RECORD SETS"
}

for ZONE in $*
do
  ###
  # if the zone does not exist (no 'id' file), create it
  ###
  
  if [ ! -f zones/"$ZONE"/id ]
  then
    echo '
<CreateHostedZoneRequest xmlns="https://route53.amazonaws.com/doc/2010-10-01/">
  <Name>'"$ZONE"'.</Name>
  <CallerReference>'`pwgen -s 16 1`'</CallerReference>
  <HostedZoneConfig>
    <Comment>aws-route-53-push</Comment>
  </HostedZoneConfig>
</CreateHostedZoneRequest>
' > "$TMPINP"
    dnscurl_post hostedzone
    cat "$TMPOUT" | egrep -o '<Id>/hostedzone/[0-9A-Za-z]*</Id>' | sed -e 's:[^/]*/[^/]*/::' -e 's:<.*::' > zones/"$ZONE"/id
  fi
  
  ###
  # read ZONEID from 'id' file
  ###
  
  ZONEID="`cat zones/"$ZONE"/id`"
  
  ###
  # determine current record sets
  ###
  
  dnscurl_obtain_rrs "$ZONEID"
  
  ###
  # update records
  ###
  
  (
    echo '
<ChangeResourceRecordSetsRequest xmlns="https://route53.amazonaws.com/doc/2010-10-01/">
  <ChangeBatch>
    <Comment>aws-route53-push</Comment>
    <Changes>'
    
    ###
    # delete before create
    ###
    
    if [ -f "$TMPRRS" ]
    then
      (
        while read -r LINE
        do
          echo
          echo '<Change><Action>DELETE</Action>'"$LINE"'</Change>'
        done
      ) < "$TMPRRS"
    fi
    
    ###
    # create after delete
    ###
    
    if [ -d zones/"$ZONE"/A ]
    then
      for EACH in zones/"$ZONE"/A/*
      do
        (
          read FQDN TTLS
          
          echo '
      <Change>
        <Action>CREATE</Action>
        <ResourceRecordSet>
          <Name>'"$FQDN"'.</Name>
          <Type>A</Type>
          <TTL>'"$TTLS"'</TTL>
          <ResourceRecords>'
          
          while read QUAD
          do
            echo '
            <ResourceRecord>
              <Value>'"$QUAD"'</Value>
            </ResourceRecord>'
          done
          
          echo '
          </ResourceRecords>
        </ResourceRecordSet>
      </Change>'
        ) < "$EACH"
      done
    fi
    
    if [ -d zones/"$ZONE"/MX ]
    then
      for EACH in zones/"$ZONE"/MX/*
      do
        (
          read FQDN TTLS
          
          echo '
      <Change>
        <Action>CREATE</Action>
        <ResourceRecordSet>
          <Name>'"$FQDN"'.</Name>
          <Type>MX</Type>
          <TTL>'"$TTLS"'</TTL>
          <ResourceRecords>'
          
          while read MAIL PRIO
          do
            echo '
            <ResourceRecord>
              <Value>'"$PRIO"' '"$MAIL"'</Value>
            </ResourceRecord>'
          done
          
          echo '
          </ResourceRecords>
        </ResourceRecordSet>
      </Change>'
        ) < "$EACH"
      done
    fi
    
    if [ -d zones/"$ZONE"/CNAME ]
    then
      for EACH in zones/"$ZONE"/CNAME/*
      do
        (
          read WCDN TTLS
          
          echo '
      <Change>
        <Action>CREATE</Action>
        <ResourceRecordSet>
          <Name>'"$WCDN"'.</Name>
          <Type>CNAME</Type>
          <TTL>'"$TTLS"'</TTL>
          <ResourceRecords>'
          
          while read DEST
          do
            echo '
            <ResourceRecord>
              <Value>'"$DEST"'</Value>
            </ResourceRecord>'
          done
          
          echo '
          </ResourceRecords>
        </ResourceRecordSet>
      </Change>'
        ) < "$EACH"
      done
    fi
    
    echo '
    </Changes>
  </ChangeBatch>
</ChangeResourceRecordSetsRequest>'
  ) > "$TMPINP"
  dnscurl_post 'hostedzone/'"$ZONEID"'/rrset'
  cat "$TMPOUT"
  echo
  CHID="`cat "$TMPOUT" | egrep -o '<Id>/change/[0-9a-zA-Z]*</Id>' | sed -e 's:[^/]*/[^/]*/::' -e 's:<.*::'`"
  while ! egrep -q '<Status>INSYNC</Status>' "$TMPOUT"
  do
    echo "waiting for CHID='$CHID' ..."
    dnscurl_get 'change/'"$CHID"
    cat "$TMPOUT"
    echo
    sleep 1
  done
  
  ###
  # finally, fetch again current record sets and list dns servers
  ###
  
  dnscurl_obtain_rrs "$ZONEID"
  cp "$TMPRRS" zones/"$ZONE"/rrs
  dnscurl_get 'hostedzone/'"$ZONEID" &> /dev/null
  echo "<========== ENTER DNS SERVER LISTING"
  cat "$TMPOUT" | egrep -o '<NameServer>[0-9A-Za-z.-]*</NameServer>' | sed -e 's:[^>]*>::' -e 's:<.*::' | tee zones/"$ZONE"/dns2
  mv zones/"$ZONE"/dns2 zones/"$ZONE"/dns
  echo "<========== LEAVE DNS SERVER LISTING"
done

I think it's not bad for a day's work.

3 comments:

  1. "Amazon Route 53 is nice good service to work with. Being developer I would like to introduce our new application, DNS30 Professional Edition.
    We have developed a UI tool for Amazon Route 53 services – DNS30 Professional Edition." http://www.dns30.com/

    ReplyDelete
  2. The main problem with third-party web-based interfaces such as DNS30 (see previous comment by DNS30 developer) is that you have to give the third-party web-based interface an access key to your AWS account. This allows hijacking of your domain by anyone who can crack the site of DNS30.

    Besides, with text files, edits are really fast ...

    ReplyDelete
  3. This comment has been removed by a blog administrator.

    ReplyDelete