[Techtalk] seeking backup solution recommendations
Meryll Larkin
mll at alwanza.com
Mon Feb 8 19:06:46 UTC 2021
Hi Conor,
Thank you for responding and for all that information!
Since my partition is formatted in the xfs file system, I should try the
xfsdump solution.
It appears to me (from my reading of the man pages, but my understanding may
be faulty) that "dump" (or xfsdump) will not permit me to select directories
or subdirectories for backup, but rather backs up an entire partition. If
that is the case, it may be less useful to me than "tar", but still worth a
try.
Thank you for your script suggestions. If I get to try it, I'll write back
and let you know how it worked for me.
No one else has responded, and, you can probably guess since I am still
using "tar", I haven't found anything better yet.
Meryll Larkin
-----Original Message-----
From: Conor Daly [mailto:c.daly at met.ie]
Sent: Tuesday, February 2, 2021 2:19 AM
To: Meryll Larkin
Cc: techtalk at linuxchix.org
Subject: Re: [Techtalk] seeking backup solution recommendations
Hi Meryll,
I know it's a while ago now but I'm wondering did you get a solution?
I have used unix Dump/Restore as a backup solution for production servers
for many years. The advantage of dump over tar is that it does things like
start a catalog at the beginning of the dump so you don't have to read the
whole tape to see what's on it. It's also cross-server retrievable and it
can do tape changes.
I think it covers all your bases...
Comments inline below
Regards,
Conor
On Sat, Nov 28, 2020 at 09:52:11AM -0800 or thereabouts, Meryll Larkin
wrote:
> Hi Systers,
>
>
>
> I'm searching online too, but I thought emailing this list might help me
as
> some of you probably have experience with different backup solutions.
>
>
>
> The environment that I am backing up has the following features:
>
> 1. Only Linux servers in the LAN- no other OS.
Yep, same here.
> 2. Only ONE DIRECTORY really requires backup. That directory is 140
T
> in size, but currently contains 25 T of data. That could double over the
> next 4 years, not sooner.
Sounds like lots of tapes. Does that have a decent sub-directory structure?
Does it have an 'archive' area that does not change as well as an 'active'
area where changes occur? This can affect how you backup. You could use a
backup plan like:
1st of month: Full backup of 'archive' area. Might take a day or
more to complete depending on drive speed.
Friday: Full backup of 'active' area. This starts when everybody
is gone for the weekend. Not too long as only 'active' are
is backed up.
Mon - Thu: Level 1 (incremental) backup of 'active' area. This is
pretty quick as it only backs up changes since last full
backup.
> 3. Some of the files in that one directory are 20G in size, and they
> could get larger.
Is it the case that a file will be actively growing and can you schedule
backups for when this is not happening?
> 4. We have a tape drive - which we have been using to make backups
> using "tar". That has reached its useful limit with those large files.
> The tapes hold about 12T of uncompressed data, each. We have several
tapes.
> We have never gotten the "Autoloader" feature to work on the tape drive,
but
> we can move the tapes manually, remotely. I'm smart about selecting
pieces
> to backup that fit on each tape.
dump has a '-F <script>' switch which specifies a script to run when a tape
fills. Because it's just a path to a script, you control what it does. In
my case, the 'swaptape.sh' script does the work on the autoloader using the
'mtx' command.
I developed a fairly simple (I hope!) configuration to manage tapes and
filesystems. It looks something like this:
###########################################################
Slot Filesystem(s)
1-3 srv3:/ srv3:/boot srv3:/home srv3:/d8 srv2:/archive
4 / /boot /home /usr /var /db_ckpt
# comment
###########################################################
Slot 1-3 indicates the backup should start on the tape in slot 1 and should
use the tape in slot 2 when tape 1 fills, moving on to slot 3 when tape 2
fills.
Slot 4 indicates the backup should use only the tape in slot 4 and will fail
if this tape fills.
srv3:/home says backup filesystem /home on remote server 'srv3'
/db_ckpt says backup the database checkpoint filesystem '/db_ckpt' on the
local machine.
Log output from the backup looks like:
Starting MASTER backups
Target: bkpsrv:/dev/nst0
Filesystem | Level | Slot | Barcode | Start date | Status
| Finish date | Barcode
----------------------------------------------------------------------------
-------------------------------
srv3:/ | 1 | 2 | 000059L5 | 2021-02-01 17:06 | Success
| 2021-02-01 17:08 | 000059L5
srv3:/boot | 1 | 2 | 000059L5 | 2021-02-01 17:08 | Success
| 2021-02-01 17:08 | 000059L5
srv3:/home | 1 | 2 | 000059L5 | 2021-02-01 17:08 | Success
| 2021-02-01 17:46 | 000059L5
srv3:/d8 | 1 | 2 | 000059L5 | 2021-02-01 17:46 | Success
| 2021-02-01 18:45 | 000059L5
srv2:/archive | 1 | 2 | 000059L5 | 2021-02-01 19:05 | Success
| 2021-02-01 19:05 | 000059L5
----------------------------------------------------------------------------
-------------------------------
Filesystem | Level | Slot | Barcode | Start date | Status
| Finish date | Barcode
----------------------------------------------------------------------------
-------------------------------
/ | 1 | 3 | 000060L5 | 2021-02-01 19:11 | Success
| 2021-02-01 19:11 | 000060L5
/boot | 1 | 3 | 000060L5 | 2021-02-01 19:11 | Success
| 2021-02-01 19:11 | 000060L5
/home | 1 | 3 | 000060L5 | 2021-02-01 19:11 | Success
| 2021-02-01 19:11 | 000060L5
/usr | 1 | 3 | 000060L5 | 2021-02-01 19:11 | Success
| 2021-02-01 19:12 | 000060L5
/var | 1 | 3 | 000060L5 | 2021-02-01 19:12 | Success
| 2021-02-01 19:12 | 000060L5
/db_ckpt | 1 | 3 | 000060L5 | 2021-02-01 19:12 | Success
| 2021-02-01 19:15 | 000060L5
----------------------------------------------------------------------------
-------------------------------
and the detailed dump output goes to a second logfile to be studied in the
case of error.
Our tape library has a barcode reader so these are logged also. The tape
change shows up as something like:
srv2:/archive | 0 | 1 | 000050L5 | 2021-01-27 21:29 | Success
| 2021-01-28 07:49 | 000059L5
where the second barcode differs from the first.
It all depends on basic unix tape tools which you will have installed to
manage your tape library:
mt: Interact with the tape in the drive
mtx: Interact with the tape library
dump: Backup ext2/3/4 filesystems
restore: Restore ext2/3/4 filesystems
xfsdump: Backup xfs filesystems
xfsrestore: Restore xfs filesystems
I did have an issue with xfsdump on large filesystems where the backup would
fail to resume on tape change which seemed to be down to a hardware
mismatch. I think it was eventually resolved.
> 5. We have an internal DNS server for the LAN only, I mention this
> because I remember a decade ago some backup solutions needed to use server
> names and have reverse lookup. We can do that.
I don't think dump requires this, it can likely do ip addresses.
> 6. We don't currently have email going in or out of the LAN. The
email
> sits in /var/spool/mail. So what that means is that if there is a GUI
> dashboard, the GUI will need to display errors. It would be helpful to be
> able to see errors some way OTHER than email notifications. Okay if I
need
> to use command line to pull them.
My solution would output to log files. Chunks of these would get emailed
but I had an active log review schedule every morning.
> 7. This is ALSO our backup and recovery plan so it is IMPORTANT that
> Restores can be done from storage media EVEN IF the original server that
> created the backup is destroyed. In other words, I need to be able to
make
> a new server, install the backup software, and restore to new hardware.
Yep, dump/restore does this. Only caveat is matching versions of
dump/restore. I cannot remember but I suspect a later restore can read
older dumps.
> I'm mentioning this because I think SOME encryption algorithms prevent the
> ability to restore from non-original backup software install.
>
> I manually retrieve tapes with backups on them so they can be
stored
> off-site.
>
> 8. Nice to have: Speed at backing up - because folks using the
> resources don't like to have their processes slowed nor access denied for
> very long.
See my 'archive' vs 'active' suggestion above.
> 9. Nice to have: Granular Restores
'restore -i' gives you a (very traditional unix) interactive session with a
tape where you can select individual files for restore.
>
>
> Suggestions? Ideas?
>
> I'd like to hear about BOTH solutions that FIT my criteria AND if you know
> solutions that DON'T (so I don't waste my time in my online research).
>
>
>
> TIA,
>
> Meryll Larkin
>
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> Techtalk mailing list
> Techtalk at linuxchix.org
> https://mailman.linuxchix.org/mailman/listinfo/techtalk
--
Conor Daly (Meteorologist),
Met Eireann, Glasnevin Hill,
Dublin 9, Ireland
Ph +3531 8064261
****************************************************************************
*****
This e-mail and any files transmitted with it are confidential and intended
solely for the addressee. If you have received this email in error please
notify the sender.
This e-mail message has also been scanned for the presence of computer
viruses.
Ta an riomhphost seo, agus aon chomhad ata nasctha leis, faoi run agus is
don te a seoladh chuige amhain e. Ma tharla go bhfuair tu an riomhphost seo
tri dhearmad cuir in iul don te a sheol e led' thoil.
Ta an teachtaireacht riomhphoist seo scuabtha le bogearrai frithvireas.
****************************************************************************
****
_______________________________________________
Techtalk mailing list
Techtalk at linuxchix.org
https://mailman.linuxchix.org/mailman/listinfo/techtalk
More information about the Techtalk
mailing list