CPCWiki backups

[Gryzor]

After CPC-Power went down some users asked me if there was enough security on the wiki and what plans are in place to restore it, if something goes awry. I was already preparing something, so I replied that I'd say it in good time, and now it's done.


As far as security itself goes, I'm no security expert. But I'm trying to be careful with both mediawiki (the Wiki platform) and SMF (forum), follow things and patch stuff whenever necessary. Regarding the server's security, I'm working with someone who is much, much more knowledgeable and am pretty confident that things are quite secure.


But, even if something happens, not much will be lost.
As of now, we have three layers of back up:
-Daily on-server db backups. This is mostly to defend against bugs, screw-ups and 0-day exploits
-Daily, complete, off-site backups. Theoretically, if the entire server went down, we could just move IPs and continue running from the backup server in a matter of hours
-Twice a week, a complete backup on my home PC (just a RAID 1, but should do).


I hope this puts you at ease


Cheers
T

[ralferoo]

Good work. I'd also suggest doing an incremental backup (maybe from your home PC), as this will provide more diagnostic / recovery options if the site is hacked and it's a little while before it's noticed. With the system as described above, anything deleted more than half a week ago but unnoticed will be lost.

[Gryzor]

Nice thought/suggestion man, thanks for pointing it out!


It's a bit convoluted but may be worth sorting out; I mean, I review changes on the Wiki on a daily (almost) basis, so if anything gets touched content-wise I see it right away. Regarding the forum, if one could get Admin access (smf exploit or su on the system/db access) theoretically it could go unnoticed. I'll look into how I could keep closer tabs on such things.


Regarding incremental backups; I'm rsyncing the thing now, which is incremental, I guess you mean *keeping* the increment history. I'm unsure how this would work - with full filesets or hardlinks?

[Bryce]

I gave it t thorough test too! I just typed "Delete CPCWiki" into Google and the Wiki is still there, so in my eyes it's safe as houses (Oh wait, since the housing crisis they aren't safe either).

Bryce.

[Gryzor]

Not only didn't you use the correct syntax on Google, but also the backup server is NOT housed in an, erm, house, but a datacenter.

[ralferoo]

Regarding incremental backups; I'm rsyncing the thing now, which is incremental, I guess you mean *keeping* the increment history. I'm unsure how this would work - with full filesets or hardlinks?

If you're using rsync (I'm assuming linux), you can specify -H (or --hard-links) to get hard-links to existing files and --link-dest to specify a compare directory. This approach is very good for files that change infrequently (especially large ones). I'll attach a copy of my personal backup script I use at home for all my computers onto a USB drive.

If you tend to have most of your data in single files, e.g. database files then rsync and hard links won't help you. However, for example with mysql databases, you can do mysqldump to get an ASCII version of the database, and if you then diff that file against the previous, you'll get the changes. It's not immediately useful for doing deltas between versions of the database, but if certainly allows you to keep the database history relatively concisely.

[Gryzor]

Yes, I'm using a Linux box though all the Windows rsync implementations I've seen use the same, proper syntax for it. but anyway. The syntax I use is a simple rsync -avvhz --delete, so as you can see it's always 'fresh'.


Thanks for the script, will certainly take a look I liked this one, with some modifications:

Code Select Expand


BACKUPDIR=`date +%A`
OPTS="--force --ignore-errors --delete-excluded --exclude-from=$EXCLUDES
      --delete --backup --backup-dir=/$BACKUPDIR -a"

export PATH=$PATH:/bin:/usr/bin:/usr/local/bin

# the following line clears the last weeks incremental directory
[ -d $HOME/emptydir ] || mkdir $HOME/emptydir
rsync --delete -a $HOME/emptydir/ $BSERVER::$USER/$BACKUPDIR/
rmdir $HOME/emptydir

# now the actual transfer
rsync $OPTS $BDIR $BSERVER::$USER/current



We do have many hundreds of thousands of files, but we also have two databases. So, with each pass rsync gets the files but also gets the latest db dump. In the past I have indeed diff'ed db backups to see what went wrong where in case of problems, but mysqlbackup also supports incremental/diff backups. So I was thinking of a cron scenario where:
-db is dumped
-an incremental db backup is done (supposedly we should take the system offline for the two to completely agree, but at 4 or 5am CET I think it's a relative minimal risk)
-rsync at backup server is started
-results are posted on a web page
-an android app polls the page for changes and reports in a nicely formatted pop-up (yet to be developped ).


Thanks for your help, much appreciated. If you got any other ideas do let me know!
T

[Gryzor]

Ok, we're now keeping backups for a number of days

[ivarf]

Ok, we're now keeping backups for a number of days

For safety, someone should be keeping several monthly full backups of the site for at least a year...

[Gryzor]

We're not the library of Congress


In the event, with hardlinks incremental backups don't take up so much space so it's going to go back for quite some time. But full monthly for a year or more? I don't think so...