Comments on: How To Create an iSCSI SAN using Heartbeat, DRBD, and OCFS2 http://www.creativeanvil.com/blog/2008/how-to-create-an-iscsi-san-using-heartbeat-drbd-and-ocfs2/ Sat, 04 Jul 2009 15:25:14 +0000 http://wordpress.org/?v=2.6.2 By: admin http://www.creativeanvil.com/blog/2008/how-to-create-an-iscsi-san-using-heartbeat-drbd-and-ocfs2/#comment-654 admin Wed, 10 Jun 2009 14:28:37 +0000 http://www.creativeanvil.com/blog/2008/how-to-create-an-iscsi-san-using-heartbeat-drbd-and-ocfs2/#comment-654 When node1 comes back up, it will come back up as secondary - as the passive server. If node2 (the current active node) were to go down, node1 would go back into primary/active mode - no manual intervention required. auto_failback keeps the system from automatically making node1 the primary/active node as soon as it returns. This is desired since OCFS will cause the server to crash if the disk disappears (which would happen briefly during failover). One thing to note - by default, a linux kernel will hang on a kernel panic (which is what happens when the disk disappears). This can be done by "echo 5 > /proc/sys/kernel/panic" that will set it to reboot after 5 seconds. This should be set on each individual node that mounts the iSCSI disk. When node1 comes back up, it will come back up as secondary - as the passive server. If node2 (the current active node) were to go down, node1 would go back into primary/active mode - no manual intervention required. auto_failback keeps the system from automatically making node1 the primary/active node as soon as it returns. This is desired since OCFS will cause the server to crash if the disk disappears (which would happen briefly during failover).

One thing to note - by default, a linux kernel will hang on a kernel panic (which is what happens when the disk disappears). This can be done by “echo 5 > /proc/sys/kernel/panic” that will set it to reboot after 5 seconds. This should be set on each individual node that mounts the iSCSI disk.

]]> By: Tan http://www.creativeanvil.com/blog/2008/how-to-create-an-iscsi-san-using-heartbeat-drbd-and-ocfs2/#comment-647 Tan Tue, 09 Jun 2009 11:53:26 +0000 http://www.creativeanvil.com/blog/2008/how-to-create-an-iscsi-san-using-heartbeat-drbd-and-ocfs2/#comment-647 Thanks for the answer. In this Active-Passive setup, when Active node (say node1) goes down because of may be OS crash or power failure, Passive node (say node2) becomes Active resuming the services. However when the node1 comes back up, since 'auto_failback' parameter is set to OFF, node2 will still act as Active. My question is - If now node2 goes down then does it fail over to node1 automatically by HA or we have to do something manually? Thanks for the answer.

In this Active-Passive setup, when Active node (say node1) goes down because of may be OS crash or power failure, Passive node (say node2) becomes Active resuming the services. However when the node1 comes back up, since ‘auto_failback’ parameter is set to OFF, node2 will still act as Active. My question is - If now node2 goes down then does it fail over to node1 automatically by HA or we have to do something manually?

]]> By: admin http://www.creativeanvil.com/blog/2008/how-to-create-an-iscsi-san-using-heartbeat-drbd-and-ocfs2/#comment-617 admin Tue, 02 Jun 2009 14:31:53 +0000 http://www.creativeanvil.com/blog/2008/how-to-create-an-iscsi-san-using-heartbeat-drbd-and-ocfs2/#comment-617 In this set-up, yes, it does lose the connection. It typically loses it for about 30 seconds, but it depends on your heartbeat configuration and how often you are monitoring it. It will lose it during the failover process. One side effect is that in OCFS, when the drive just disappears, it causes a kernel panic. I set the boxes in the cluster to reboot on a kernel panic. By the time the boxes reboot, the new drive (iSCSI) is back online and services resume as normal. So, it's less than ideal, as it's not a fully transparent failover. However, in my case, having 2-3 minutes of downtime was acceptable. If I didn't have the redundancy in the SAN and I lost drives, I'd be totally down. For my setup, I needed 1TB of space, and at the time I wrote this, this was the most cost effective way to achieve 1TB of shared, clustered storage. I'm willing to bet a setup like this is still the most cost effective, if you can afford the few minutes of downtime that may occur. Good question, thanks for asking. In this set-up, yes, it does lose the connection. It typically loses it for about 30 seconds, but it depends on your heartbeat configuration and how often you are monitoring it. It will lose it during the failover process. One side effect is that in OCFS, when the drive just disappears, it causes a kernel panic. I set the boxes in the cluster to reboot on a kernel panic. By the time the boxes reboot, the new drive (iSCSI) is back online and services resume as normal. So, it’s less than ideal, as it’s not a fully transparent failover. However, in my case, having 2-3 minutes of downtime was acceptable. If I didn’t have the redundancy in the SAN and I lost drives, I’d be totally down. For my setup, I needed 1TB of space, and at the time I wrote this, this was the most cost effective way to achieve 1TB of shared, clustered storage. I’m willing to bet a setup like this is still the most cost effective, if you can afford the few minutes of downtime that may occur. Good question, thanks for asking.

]]> By: Tan http://www.creativeanvil.com/blog/2008/how-to-create-an-iscsi-san-using-heartbeat-drbd-and-ocfs2/#comment-615 Tan Tue, 02 Jun 2009 08:32:52 +0000 http://www.creativeanvil.com/blog/2008/how-to-create-an-iscsi-san-using-heartbeat-drbd-and-ocfs2/#comment-615 At the time of failover, does the iSCSI connection (session) between iSCSI initiator and iSCSI target running on master server break? If it breaks, then does initiator re-initiates the connection? How much is the downtime for IO in that case? At the time of failover, does the iSCSI connection (session) between iSCSI initiator and iSCSI target running on master server break? If it breaks, then does initiator re-initiates the connection? How much is the downtime for IO in that case?

]]> By: admin http://www.creativeanvil.com/blog/2008/how-to-create-an-iscsi-san-using-heartbeat-drbd-and-ocfs2/#comment-340 admin Thu, 02 Apr 2009 19:23:49 +0000 http://www.creativeanvil.com/blog/2008/how-to-create-an-iscsi-san-using-heartbeat-drbd-and-ocfs2/#comment-340 Yes and no, it's important to have 2 different connects between the boxes for the 'heartbeat' to exist on. In my case, it's a serial cable and the network connection. The network connection is just a crossover cable, so you would have to have a bad cable or NIC for that to go out. In the case it does (certainly could), the heartbeat checks over the serial connection to see if the other server is still alive. Now, if both connections fail, yeah, you're right, that could happen. You should look into STONITH for that reason. Check out http://linux-ha.org/STONITH. Hope this helps. Yes and no, it’s important to have 2 different connects between the boxes for the ‘heartbeat’ to exist on. In my case, it’s a serial cable and the network connection. The network connection is just a crossover cable, so you would have to have a bad cable or NIC for that to go out. In the case it does (certainly could), the heartbeat checks over the serial connection to see if the other server is still alive. Now, if both connections fail, yeah, you’re right, that could happen. You should look into STONITH for that reason. Check out http://linux-ha.org/STONITH. Hope this helps.

]]> By: Matthew http://www.creativeanvil.com/blog/2008/how-to-create-an-iscsi-san-using-heartbeat-drbd-and-ocfs2/#comment-339 Matthew Thu, 02 Apr 2009 19:11:06 +0000 http://www.creativeanvil.com/blog/2008/how-to-create-an-iscsi-san-using-heartbeat-drbd-and-ocfs2/#comment-339 I'm looking at a setup similar to this, and one thing worries me. What happens if you lose the connection between the two servers? Presumably the primary server will keep running, and the secondary will consider the primary dead and try to take over.This will leave both boxes claiming the single IP address and export the DRBD volume, leading to chaos. Any thoughts on this situation greatly appreciated! I’m looking at a setup similar to this, and one thing worries me. What happens if you lose the connection between the two servers? Presumably the primary server will keep running, and the secondary will consider the primary dead and try to take over.This will leave both boxes claiming the single IP address and export the DRBD volume, leading to chaos. Any thoughts on this situation greatly appreciated!

]]> By: admin http://www.creativeanvil.com/blog/2008/how-to-create-an-iscsi-san-using-heartbeat-drbd-and-ocfs2/#comment-163 admin Mon, 29 Dec 2008 15:17:56 +0000 http://www.creativeanvil.com/blog/2008/how-to-create-an-iscsi-san-using-heartbeat-drbd-and-ocfs2/#comment-163 Yeah, I pulled the power cord during a write. No major bad things happened. How were you using FAT32 and NTFS as the filesystem though? I was using OCFS2 since it's an actual cluster file system. If you don't have a cluster-aware file system, bad things certainly will happen, including a lot of corruption. I'm not aware that NTFS would work for this, but I could be wrong. Yeah, I pulled the power cord during a write. No major bad things happened. How were you using FAT32 and NTFS as the filesystem though? I was using OCFS2 since it’s an actual cluster file system. If you don’t have a cluster-aware file system, bad things certainly will happen, including a lot of corruption. I’m not aware that NTFS would work for this, but I could be wrong.

]]> By: Anon http://www.creativeanvil.com/blog/2008/how-to-create-an-iscsi-san-using-heartbeat-drbd-and-ocfs2/#comment-161 Anon Sun, 28 Dec 2008 23:23:18 +0000 http://www.creativeanvil.com/blog/2008/how-to-create-an-iscsi-san-using-heartbeat-drbd-and-ocfs2/#comment-161 Did you test the failover when a write was occuring? Me and a friend of mine were testing a similar setup using windows/ubuntu clients to mount an iscsi drive shared between 2 drbd nodes. Provided we were using FAT32 and ntfs as the target filesystem, we noticed that if the primary drbd node was shutdown in the middle of a large write, we would get data corruption. We were also using the provided iscsi target daemon (tgt, or scsi-target-utils in centos 5.2) We were thinking this was either a problem with the iscsi daemon not commiting the writes to disk (as there was a large chunk of data missing in the file for when the failover occurred). Was there a specific reason you used ocfs2 as the filesystem? Thanks for the writeup! Did you test the failover when a write was occuring? Me and a friend of mine were testing a similar setup using windows/ubuntu clients to mount an iscsi drive shared between 2 drbd nodes. Provided we were using FAT32 and ntfs as the target filesystem, we noticed that if the primary drbd node was shutdown in the middle of a large write, we would get data corruption. We were also using the provided iscsi target daemon (tgt, or scsi-target-utils in centos 5.2)

We were thinking this was either a problem with the iscsi daemon not commiting the writes to disk (as there was a large chunk of data missing in the file for when the failover occurred). Was there a specific reason you used ocfs2 as the filesystem?

Thanks for the writeup!

]]> By: admin http://www.creativeanvil.com/blog/2008/how-to-create-an-iscsi-san-using-heartbeat-drbd-and-ocfs2/#comment-10 admin Thu, 23 Oct 2008 21:45:59 +0000 http://www.creativeanvil.com/blog/2008/how-to-create-an-iscsi-san-using-heartbeat-drbd-and-ocfs2/#comment-10 Sorry for the massive delay in a response. I wasn't receiving notifications when comments were posted. I'm sure it's too late for you, but for the sake of others... the setup was about $10,000 at the time, though I'm sure you could do it for less now. Each of the SAN servers was almost $4,000 when I bought them. I think you could use 1U servers now with RAID 1 and still easily get 1TB, which was my goal. The problem I had with the Dell SANs was the cost was about $7,000 for a simple iSCSI SAN that didn't provide any redundancy at all. I don't really think it'd be possible to just start with one server for the SAN portion. You'd have to later add DRBD and heartbeat, which, I believe could be done, but it would be pretty disruptive. Sorry for the massive delay in a response. I wasn’t receiving notifications when comments were posted. I’m sure it’s too late for you, but for the sake of others… the setup was about $10,000 at the time, though I’m sure you could do it for less now. Each of the SAN servers was almost $4,000 when I bought them. I think you could use 1U servers now with RAID 1 and still easily get 1TB, which was my goal.

The problem I had with the Dell SANs was the cost was about $7,000 for a simple iSCSI SAN that didn’t provide any redundancy at all.

I don’t really think it’d be possible to just start with one server for the SAN portion. You’d have to later add DRBD and heartbeat, which, I believe could be done, but it would be pretty disruptive.

]]> By: Marcus http://www.creativeanvil.com/blog/2008/how-to-create-an-iscsi-san-using-heartbeat-drbd-and-ocfs2/#comment-9 Marcus Sun, 03 Aug 2008 00:37:00 +0000 http://www.creativeanvil.com/blog/2008/how-to-create-an-iscsi-san-using-heartbeat-drbd-and-ocfs2/#comment-9 Oh, another question. Is it possible to start with just one 2950 and then expand to two without a huge interruption to service. Or does DRBD require configuration before creating the volumes/etc? Oh, another question. Is it possible to start with just one 2950 and then expand to two without a huge interruption to service. Or does DRBD require configuration before creating the volumes/etc?

]]>