Simplest way to back up a four node cluster? [Archive] - Glock Talk

PDA

View Full Version : Simplest way to back up a four node cluster?


betyourlife
05-17-2010, 23:22
What's the simplest way to back up a four node cluster? Would conducting a full backup of each node in the cluster suffice? I mean could you restore a full backup on a corrupted node and be back to where you were? In theory all the settings and such are the same.

kc8ykd
05-18-2010, 13:12
what kind of cluster is it? like, web servers or something else?

also, what operating system? and, is it some place where a human would have access to them, or a remote site. also, how much data are you talking about?


in theory, an image of one would be applicable to the others, assuming all the hardware and related settings were exactly the same os-wise, although, even then you might run into issues if it were running windows server.

Pierre!
05-18-2010, 19:07
I would tend to agree with kc8ykd.

Most "clusters" am hearing about today are sharing the same data off a SAN... but I do recall back when clusters of identical equipment mirrored to each other.

What's the goal (RPO/RTO) of the exercise, and what kind of $$$ do you have to spend?

That will color the answer... *GREATLY*

so... more details please!

kc8ykd
05-18-2010, 20:31
this is GT, where $$ has no meaning ! (especially when it's someone else's money)

we used to have a saying at my old company, "problem? throw money at it until it's fixed" worked great in theory, not so great in practice :(

JK-linux
05-18-2010, 20:47
.....

Linux3
05-19-2010, 05:59
Fun stuff. Simplest of best way?
To get the ball rolling I'll make some assumptions based on the clusters I have built.
We will assume you are using Red Hat Linux or CentOS or Fedora.
We will assume you built your nodes using a "kickstart server".
http://centos.org/docs/5/html/Installation_Guide-en-US/s1-kickstart2-howuse.html

Thus you can do a bare metal replacement of any node quite easily so we are talking a backup of data.
I assume each node has a O.S. disk and a data disk. Well, we've covered the O.S. disk.
Data, here's where we need more info. If running MySQL then use the built in functions to back up your records and transactions among all the nodes via NFS mount. Master server and 3 slaves configuration.

If something else I have used GFS2, Global File System2, and made the nodes into a RAID via NFS.
http://en.wikipedia.org/wiki/Global_File_System

Most likely none of this pertains to you. See what others and I have been saying. We need more info.
Not that this hasn't been a fun exercise.

betyourlife
05-23-2010, 22:06
what kind of cluster is it? like, web servers or something else?

also, what operating system? and, is it some place where a human would have access to them, or a remote site. also, how much data are you talking about?


in theory, an image of one would be applicable to the others, assuming all the hardware and related settings were exactly the same os-wise, although, even then you might run into issues if it were running windows server.

Application and database servers. Let's say it's a Windows Server OS (2003/2008)

The only thing that's large on the disk other than the OS would be the database.

It seems that you (as well as my original assumption) are right, according to this article, backing up any of the nodes in the cluster would suffice.

http://searchdatabackup.techtarget.com/tip/0,289483,sid187_gci1415397,00.html

I guess the key is to make sure I either have an exact spare of a node in the cluster handy in case it is related to catastrophic hardware failure, OR make sure I can get one damn fast.

kc8ykd
05-24-2010, 01:56
another option might be a hot spare server that you can rdp into to activate, or if you're using a content redirector, tell the redir to shift traffic from the dead server to the hot spare.

as for the database, you should be able to freeze it and dump all the data depending on how it's configured.

my experience is more with a setup that includes a redirector, servers, and a netapp on a private lan connected to the servers (each server had a dual-nic). all the data was stored on the netapp which could perform snaphot backups on itself and the backups of the servers went to the netapp. the redirector was a device and not a computer, so we just backed up it's config to a tftp machine we had running for such purposes.

we had another system that was a lot more complex, but similar, which the redirectors were actual computers that used ospf to talk to the content servers for availability and content, and the redirs could reload an entire content server using netboot if necessary. they also had watchdog cards in them that would reboot the system automatically if they os hung for a certain amount of time. this also used another netapp for content storage. I think that system used freebsd for the OS (we never did much with windows for servers at all).

the intricate details i don't really know since i managed all the network stuff and our software guys did all the actual server stuff. all the servers ran some form of linux, either debian or centos and occasionally freebsd. my knowledge of the software side was just enough so i could help them troubleshoot issues (so i could rule out my network as the problem, basically).

the netapps we used were a hybrid nas/san. nas in that they communicated to the various servers via private 802.3u networks using NFS or CIFS (for the odd windows server), san in that the filer talked to the disk shelves using fibrechannel.

JK-linux
05-24-2010, 05:59
.....