Automated offsite backups

Discussion in 'General' started by Chilling_Silence, Jun 15, 2010.

  1. Chilling_Silence

    Joined:
    Sep 23, 2008
    Messages:
    488
    Likes Received:
    0
    Hi all,

    I'm looking for some sort of way to easily, automatically and remotely backup everything on some of my Elastix-1.6 systems.

    Ideally I'd like to backup the CDRs and system recordings as well as the general configuration.

    I understand there's the backups module in the Web UI but I'm after something that can be automated.
    Any ideas?

    Thanks


    Chill.
     
  2. dicko

    Joined:
    Oct 24, 2008
    Messages:
    4,099
    Likes Received:
    0
    I use something like this:
    ----------------------------------------------------------------
    #!/bin/bash
    # set up some variables

    BUSERVER=backerupper@1.2.3.4
    BUDIR=/data/clientbackups
    DOW=`date +%A`
    DAILYDAYS=8
    ISODAYS=8
    MONITORDAYS=30
    LOCALBUDIR=/backup
    VMDAYS=90

    echo "$HOSTNAME Backup on $DOW `date`"


    ssh -p 2222 $BUSERVER mkdir -p $BUDIR/$HOSTNAME


    case $DOW in
    Sunday)

    echo "It's $DOW, lets do the big one."
    echo "Begin mondoarchive to $LOCALBUDIR at `date`"
    /usr/sbin/mondoarchive -OVi -d $LOCALBUDIR -E "/backup /var/lib/asterisk/backups /var/spool/asterisk /usr/src " -s 4800m -p "`hostname`_`date +%Y.%m.%d.%H.%M`"
    echo "End mondoarchive at `date` "
    ;;
    esac




    echo "Begin rsync at `date` to $BUSERVER:$BUDIR"
    /usr/bin/rsync -at --inplace --progress -e 'ssh -q -p 2222' $LOCALBUDIR/*.iso $BUSERVER:$BUDIR/`hostname`/
    echo "On remote server"
    /usr/bin/rsync -at --inplace --progress -e 'ssh -q -p 2222' $BUSERVER:$BUDIR/$HOSTNAME/daily/
    /usr/bin/rsync -at --inplace --progress -e 'ssh -q -p 2222' /var/lib/asterisk/backups/daily/*.gz $BUSERVER:$BUDIR/$HOSTNAME/daily
    echo "End rsync at `date` to $BUSERVER:$BUDIR"
    rsync -e 'ssh -p 2222' $BUSERVER:$BUDIR/`hostname`/*.iso
    #delete old backup files over 8 days
    #find /var/lib/asterisk/backups/daily/*.gz -mtime +$DAILYDAYS
    rsync -e 'ssh -p 2222' $BUSERVER:$BUDIR/`hostname`/daily/

    echo "Will delete"
    echo "`find /var/lib/asterisk/backups/daily/*.gz -mtime +$DAILYDAYS -delete`"
    echo "backupfiles"
    find /var/lib/asterisk/backups/daily/*.gz -mtime +$DAILYDAYS -delete

    #delete old fax temp files over $ISODAYS days old
    #find /var/www/html/fax/tmp/* -mtime +$ISODAYS -delete

    #delete monitor files older than $MONITORDAYS
    #find /var/spool/asterisk/monitor/*.wav -mtime +$MONITORDAYS -delete

    #find /var/spool/asterisk/voicemail/*/*/INBOX/* -mtime +$VMDAYS -delete

    /var/lib/asterisk/bin/ampbackup.pl daily yes yes yes yes yes

    ----------------------------------------------------------------------

    kind of 2 level, (iso's once a week, the rest daily)


    a little unpolished and not much error detection but it fits my particular environment, I hope you get the gist

    (run as a cron job)

    I also have for backing up from a working server to a semi-active "standby server", (run from the standby)
    ------------------------------------------------------
    #!/bin/bash
    service hylafax stop
    service asterisk stop
    service mysqld stop

    rsync -av --progress --delete -e 'ssh -p 2222' 1.2.3.4:/var/www/ /var/www/
    rsync -av --progress --delete -e 'ssh -p 2222' 1.2.3.4:/var/lib/mysql/ /var/lib/mysql/
    rsync -av --progress --delete -e 'ssh -p 2222' 1.2.3.4:/var/spool/hylafax/ /var/spool/hylafax/

    service mysqld start
    service hylafax start
    service asterisk start



    rsync -av --progress --delete -e 'ssh -p 2222' 1.2.3.4:/tftpboot/ /tftpboot/
    rsync -av --progress --delete -e 'ssh -p 2222' 1.2.3.4:/var/lib/asterisk/backups/ /var/lib/asterisk/backups/
    rsync -av --progress --delete -e 'ssh -p 2222' 1.2.3.4:/var/spool/asterisk/ /var/spool/asterisk/

    -------------------------------------------------------------------

    one will need to do the ssh keys thing though,

    for that I stole/hacked

    -------------------------------------------------------

    #!/bin/sh

    KEY="$HOME/.ssh/id_dsa.pub"

    if [ ! -f ~/.ssh/id_dsa.pub ];then
    echo "private key not found at $KEY"
    echo "* please create it with "ssh-keygen -t dsa" *"
    echo "* to login to the remote host without a password, don't give the key you create with ssh-keygen a password! *"
    exit
    fi

    if [ -z $1 ];then
    echo "Please specify user@host.tld as the first switch to this script and port as second switch"
    exit
    fi

    echo "Putting your key on $1... "

    KEYCODE=`cat $KEY`
    ssh -p $2 -q $1 "mkdir ~/.ssh 2>/dev/null; chmod 700 ~/.ssh; echo "$KEYCODE" >> ~/.ssh/authorized_keys; chmod 644 ~/.ssh/authorized_keys"

    echo "done!"

    ------------------------------------------------------------------------------------------




    regards

    dicko
     
  3. Chilling_Silence

    Joined:
    Sep 23, 2008
    Messages:
    488
    Likes Received:
    0
    Awesome that looks like a great starting point, thanks!

    I'll give it a whirl and see how I go, much appreciated :)
     
  4. dicko

    Joined:
    Oct 24, 2008
    Messages:
    4,099
    Likes Received:
    0
    Chill:

    If it basically works for you, please tidy up all my crap and blog it as you do so well, (I'm a sloppy lazy slob ;) )

    dicko
     
  5. Chilling_Silence

    Joined:
    Sep 23, 2008
    Messages:
    488
    Likes Received:
    0
    Sure mate!

    I've got a fair bit on today, so I probably won't be able to take a look at it immediately, but stay tuned for a blog post on it ;)

    Thanks again :)
     
  6. Chilling_Silence

    Joined:
    Sep 23, 2008
    Messages:
    488
    Likes Received:
    0
    Why stop asterisk etc though ?
     
  7. dicko

    Joined:
    Oct 24, 2008
    Messages:
    4,099
    Likes Received:
    0
    Don't forget that is run from the "standby" so shouldn't be too impactful, and was actually a transitional script as I move to drbd/pacemaker (which is NOT EASY to get going on current Elastix kernels, you guys should stick with heartbeat for now) , to be run from the earstwhile "new" production server, and I hate "down-time".

    There is a slight possibility of "collision" especially with the mysql server, so just for the hell of it I stop the perhaps problematic services, Marie Koreen (I hope I spelled her name right ) does the mysqldump more rigourously, in her expansion of the Elastix backup process.

    You will notice that the longer rsync processes are generally less critical so I moved them outside of that temporal stasis.

    I personally add mysql "jounalling" to mysqld :-

    log-bin = /var/lib/mysql/bin.log

    to /etc/my.cnf , this after being "once-bitten", this will allow an orderly rebuild from the bin-log after an eff-up, as I said, I'm a sloppy, lazy bastard :)

    dicko
     
  8. Chilling_Silence

    Joined:
    Sep 23, 2008
    Messages:
    488
    Likes Received:
    0
    Why not use something like:
    mysqldump --add-drop-table -h localhost -uroot -peLaStIx.2oo7 -A > /root/database.sql

    Then just backup / rsync that?
    Means you don't have to take anything offline... Ideally I'd like to keep the systems up 24/7, as this will be for a 24/7 support helpdesk :-/

    Or have I missed something ?
     
  9. dicko

    Joined:
    Oct 24, 2008
    Messages:
    4,099
    Likes Received:
    0
    Perhaps you did miss something, I do that all the time :)

    As stated I concur, never take down a working server unless you have to, 5 nines is what I aspire to (or better yet 60/60/24/7/52) , I never stopped the production server, just the "hot-spare" and that for just a few seconds, and restoring a mysqldump can take quite a time on some of my legacy +five year old boxes astriskcdr can run to gigabytes, it's a matter of expediency and this process plus the log-bin journal just works quicker, yet still covers your ass, (I'm also impatient ;) , but please go the safer slower route if you prefer).

    In my experience a catastrophic single server failure (thuggishly pulling the plug) in a well designed HA cluster will recover in much less than 30 seconds (which will never be noticed in a real-world "damn I mis-dialed, I'll do it over" scenario, and a couple of dropped calls in progress ) thusly your customer "pissed-off level" will approach zero. Ethically you get to get one failure every two months to maintain that 5x9, I can live with that, next step a fail-safe BGP solution to match and finding a VSP or three that also achieve that 5x9 criterion. But I digress . . .

    dicko
     
  10. Chilling_Silence

    Joined:
    Sep 23, 2008
    Messages:
    488
    Likes Received:
    0
    Aside from what's in /etc/asterisk and also /var/spool/asterisk/ if I'm also dumping MySQL, is there anything that's going to change from a vanilla Elastix install if I'm purely sticking to editing things in FreePBX?

    It's nice to do ISO's and all, but potentially overkill if I'm just wanting to backup configuration files so that the system can be restored (Overwriting a vanilla Elatix install) and also if the client makes a boo boo, we can go back to yesterdays configs.

    I'm doing some testing now, hope to blog it all shortly :)
     
  11. dicko

    Joined:
    Oct 24, 2008
    Messages:
    4,099
    Likes Received:
    0
    Overkill will save you're ass when a lazy underkill just wouldn't cover ones lack of prescience and you forgot to cover all your bases, don't be complacent, it's just cpu cycles.

    What else changed is totally dependent on YOUR system and how much it is used,


    find / -mtime -1


    will tell you "what's new today for me", if anything looks pertinent then it probably is, and thus a good case to "back it up".


    If you can currently do a "bare metal restore" from last week, and apply last night's state within an hour, then please share your recipe. My methodology does :) :)




    dicko
     
  12. dicko

    Joined:
    Oct 24, 2008
    Messages:
    4,099
    Likes Received:
    0
    For immediacy, but with some concern about the abilities of the local warm-ware, you can further reduce downtime by doing the mondoarchive to a "local" usb stick, just maintain control of both the local BIOS boot settings, and any warm-ware involved.

    dicko
     
  13. Chilling_Silence

    Joined:
    Sep 23, 2008
    Messages:
    488
    Likes Received:
    0
    Not quite using your scripts but check the blogs for my new post and let me know what you think.

    In other news the new Forums work brilliantly on my Nexus One!!
     
  14. dicko

    Joined:
    Oct 24, 2008
    Messages:
    4,099
    Likes Received:
    0
    Chilling:

    It certainly looks functional, I like the "diff" bit

    Personally I would have left the SSHPORT option in, I don't advise anyone leaving SSH on 22. I would also have left the key exchange as a script for the noobs.

    Editing amportal.conf to suit (forr rsync or ftp even) and

    /var/lib/asterisk/bin/ampbackup.pl daily yes yes yes yes yes

    and adding the cron job is also effective if you just need the core stuff.

    regards

    dicko
     
  15. Chilling_Silence

    Joined:
    Sep 23, 2008
    Messages:
    488
    Likes Received:
    0
    Yes perhaps the port could be useful, however it makes no difference if you change the SSH port on the PBX, only the port on the server everything is backing up to, but it's still a good point and a simple addition.

    What does that ampbackup script do that mine doesn't?
    Do you adjust /etc/amportal.conf on most of your systems?
     
  16. dicko

    Joined:
    Oct 24, 2008
    Messages:
    4,099
    Likes Received:
    0
    :) :)


    ALL my machines run SSH on "another port", and all for the very same reasons.

    amportal.conf, yes I edit to default new extensions to call waiting, add SSHPORT=2222 or whatever to keep the status display functional (hopefully green). In general much of FreePBX's behavior can be "tuned" here-within.

    it's pretty well self-documenting as to how to do the off site backups, adding directories, like /tftpboot, and more. the last few lines need changing to suit a drbd/heartbeat type deployment to not use sym-linked directories. I don't use the internal scp/sftp bits though, as I prefer my scripts :)

    so I guess essentially the only thing can do that yours doesn't is back-up the "complete" state of the FreePBX machine including dahdi and also itself, true, at the cost of some time, disk space, bandwidth and CPU cycles. You could add them though.

    If you look at one of the backups in /var/lib/asterisk/backups/<set>/ you will see what's included.

    You can of course also schedule on a more granular and time sensitive fashion multiple subsets of "yes yes etc." to suit the dynamics of your systems. JM2CWAE

    regards

    dicko
     

Share This Page