Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 


Forgot your password?
Close
typodupeerror
×
Linux Business

Linux Failover? 203

Posted by Cliff from the is-this-a-real-problem? dept.
Anton asks: "This is a question about Linux failover in business situations. We are a growing B2B company; our product runs on Apache/Linux. We contracted professional services to properly set up our network. After all the hellishly expensive CISCO hardware had been set up it turns out that for our servers to be configured for failover, each one needs two dual-port NICs configured for one IP connected to two different switches, furthermore the driver needs to intelligently switch the ports when the active port fails ... We've never heard of such beasts for Linux and a net search revealed nothing. Our consultant however claims that 'Linux is biting itself in the foot' for not supporting that, and that other industrial strength OS's like solaris in fact do support this. Has anyone run into this before or have other ideas? " [nik suggests]: Take a look at Polyserve Understudy, which might be an alternative. FreeBSD and Linux versions are available (and bundled with FreeBSD 4.0).
This discussion has been archived. No new comments can be posted.

Linux Failover? More

Linux Failover?

Comments Filter:
  • The old fashioned way of doing it is to set your availability requirements, then pick a solution. Many people have unrealistic expectations or unclear availability goals. The usual conversation goes:

    You: What availability do you want?
    Client: 100% - It cannot ever go down at all or the world ends
    You: Right, that'll be 10 million a year
    Client: How about we schedule some downtime every week?

    Be careful of references to 'industrial strength' operating systems. There are some very good products available but they are not part of the base OS, they can cost serious money and manufacturers will make you go through hoops to deploy them.

    For example AFAIK SunCluster configurations are built and validated by Sun and then shipped to you. They are also only available on cluster certified hardware models. If you want to run non standard software on them they will probably want that signed off as well.

    You do not automatically get application failover unless the application supports it in that particular cluster environment. Commercial cluster products tend to be biased towards transaction processing and database environments (because that's where the need and the big money is) and may not be the best option for web serving etc.

    If you want more details the enterprise computing sites of all the big manufacturers have a stack of three hundred page white papers that conclusively prove they are all the best at everything :-)

    Posting anon again because Slashdot never seems to send email to Demon users.

    Martin (martin@nospam.sharrow.demon.co.uk)

  • Ethernet Link Protector (Score:1)

    by Anonymous Coward
    For a government project requiring network redundancy I have used the following ethernet link protector. http://www.shoremicrosystems.com/sm-2500.htm It's two outputs run to two seperate 3com switches. It also has a output contact to tell you when it has failed over.

  • Linux Journal Article with HA solution... (Score:1)

    by Anonymous Coward
    I had a similar problem a few years ago and wrote some scripts to do failover clustering.
    I published it in the Linux Journal last year:
    http://ww w.linuxjournal.com/cgi-bin/frames.pl/lj-issues/iss ue64/3247.html [linuxjournal.com]
    source/scripts:
    ftp://ftp.ssc.com/pub /lj/listings/issue64/3247clusterd.tar.gz [ssc.com]
    It would solve your problem with layer-2 and layer-3 failover but, without modification, would require two identical servers.
    Yes, it will decide which switch/router is actually still live for the next hop routing aswell - it can determine which switch has failed if any by pinging a list of 'supposed to be reachable' addresses.
    It is a bit dated and I haven't really kept the code upto date but the principles are there and it is has been used for a long time with no major problems.

    Philip J Lewis
    Network Consultant
    Dome Computer Consultants Ltd
    UK
    mailto:slashdot@*REMOVEME*linuxcentre.net

  • Ramix (Score:1)

    by Anonymous Coward
    What type of network connection do you have? Assuming you are using fast ehternet Ramix makes a dual port 100BFX NIC with hardware failover. I don't know if they have Linux drivers but they definatly work with other UNIX platforms.
  • This stuff is great. I've heard about these before, but I'd have given my right arm to have been able to track these down last year. I like the software solutions, but I'd definetely like to use these, too.

    -Peter
  • Here's a configuration that should work.

    1) Compile the kernel with the bonding device as a module.
    2) Set your ethernet cards up through a bond device.
    3) Plug NIC #1 into switch #1, and NIC #2 into switch #2. Let's say they're on vlan 100.
    4) Trunk vlan 100 between the 2 switches.
    5) Set your spanning tree config to favor devices seen through switch 1 (your primary).
    6) Activate the bonding device. Once you do this, both switches will see the same MAC address, and switch #2's spanning tree config should cause it to administratively shut down the port that is connected to NIC #2.

    What happens next is that all traffic goes through NIC #1, and switch #1. However, if switch #1 goes down, or if NIC #1 goes down, then switch #1 will surrender to switch #2 and allow switch #2 to bring up NIC #2, which will share the same IP address and MAC address as NIC #1 (read the doco for the bonding driver if you don't understand how this works).

    Actually, this doesn't even require trunking the vlan, but it does require that you have switches that have spanning tree, that you pay careful attention to the spanning tree config, and that both interfaces can see each other so the spanning tree config will isolate them on the network.

    NOTE!!! YMMV I don't know for sure that this works under linux. I know that this config should work with sun trunking, and reading the bond_xmit() function in the bonding driver it looks like it should work. I will test it on some cisco equipment if I get a chance.

    If you want me to help you set this up, send me email, as I'm available to do such work.

    -Peter
  • and, FWIW, I doubt simple channel bondig would help, either

    Actually, this should be doable with channel bonding (where the MAC address is identical across all interfaces) combined with a conservative spanning tree configuration (having trunking between both switches would be sugar on top, in case of the upstream interface dying on the switch).

    See my other post for a likely configuration.

    -Peter

  • And bay and ipivot. It works, and it's a publicly available RFC. Much nicer situation then HSRP.

    -Peter
  • You're out of date as of solaris 2.6. The sun trunking module (cost of about $1000 per system it's on), bad documentation and all, takes care of this problem far better then any set of scripts can.

    Linux's bonding driver addresses this, too.

    -Peter
  • Have you ever talked to someone who has had sun consultants come in and try to do a sun cluster install? Maybe they've gotten better since last year, but I talked to at least 2 companies that had been over-promised a working solution by sun. One company ripped sun's work out of their data center (sun didn't have the cluster working a couple of months into the project, and the 3rd party worked far easier and faster) and the other settled for a reduced spec.

    *BLECH*

    -Peter
  • You can avoid the problem of having to share the channel between switches by using spanning tree to shut one off until it's needed. This ensures that only one interface and switch pair is active for a particular bonded address at any one time, eliminating the need for the switches to have to load-balance between those 8 channels that they have available.

    This solution expands to an arbitrary number of switches and an arbitrary number of host interfaces.

    -Peter
  • Really? The following code looks pretty foolproof (from bonding.c, bond_xmit()):

    while (good == 0) {
    slave = queue->current_slave->dev;
    if (slave->flags & (IFF_UP|IFF_RUNNING)) {
    skb->dev = slave;
    skb->priority = 1;
    dev_queue_xmit(skb);
    good = 1;
    }
    if (queue->current_slave->next != NULL) {
    queue->current_slave = queue->current_slave->next;
    } else {
    queue->current_slave = queue->head;
    }
    }

    Care to explain where the problem lies in detecting if the card is active or not, and how this is less reliable then an IGP?

    -Peter
  • The bonding driver does this for you w/o needing much/any intervention from the user.

    -Peter
  • Have you priced out a pair of fibre channel arrays, a fibre channel extender (over dark fibre or oc3) and a raid 1 driver? It'll be slow as shit (speed of light for the data to travel through the fibre channel->WAN->fibre channel) but it'll get all of your content somewhere else in real time. It'll probably cost you as much as an EMC box, though, which does it a bit more... sanely.

    -Peter

  • Right, it isn't the best design. However the redundant machines are designed so we can operatue without them for several minutes if need be. The recovery procedure if the Master really does fail involves the backup rebooting. (These machine control other hardware that must work all the time, and that hardware is both more robust and can operate without the controllers for a short time if need be. You just lose access to the disc so you cannot reconfigure them.

    I didn't want to get into all the head aches we faced due to the bad design above, it is byond the scope of the orginal question. We are however re-doing things to fix that.


  • Uh, your knowledge has been shot in the foot. You dont need a 3rd party app for failover in Solaris. Sun offers Sun Cluster (current version is 2.2, 3.0 is going to be released next month for Solaris 8)

    There is also Veritas First Watch.
  • Yea... Those boxes are really nice. The company I work for sells them. You gotta dig that big red ball on the front.
  • They are that expensive, because people with those kinds of needs are willing to pay that much money.

    It's the same everywhere, where have you been?

    I just remembered this old Metallica song. . .
  • For rel NIC failover, you need to have 2 cards (or more) or a dual port card. The OS must support changing the IP address of the 2nd card and reprogramming it's MAC address so the router will see it as the first card. Some routers don't like to see the same IP from a different port with a different MAC address. A real world example would be 2 Cisco Catalyst plugged together. And each computer having one card on each Catalyst. Each Catalyst will have also a link to the outside. Unless the link between the 2 Catalyst is down AND one of the card is down everything will work fine.

    SCO UnixWare does support NIC failover (for cards that support MAC address reprogramming), unfortunately it's not one of the most stable OS that I saw.

    Sometimes the same computer must complete the transaction, using another one is not an option (telephony come to mind).

  • One way to solve this is to run a Layer3 routing protocol on the box(s). No need for fancy/overblown link failover software, when you can take advantage of the reason why routing protocols were invented! Routing protocols by design route around link failures.

    Two Unix based 'routing' daemons are:

    Alternatively, assuming you've eliminated another single point of failure by running two routers, is to run HSRP ( since you mentioned Cisco ) on the routers. You could set up the network in such a way that you can dual home each server to a separate switch, without needing link failover or even a routing protocol.

    There are several ways to kill this "problem", but the way your *ahem* consultant is recommending sounds like the most overly complex solution. Your consultant may know their systems/lan configuration, but shows little knowledge of routing.

    My 2 cents is get a new consultant.
    ------

  • >Oh, that was a good joke. Yes, Nokia uses it. >And nobody else

    You are talking complete bollocks, just ask Bay, Alteon etc. VRRP works, easy to configure & a lifesaver operationally.

    >But of course they didn't even bother to >contribute it back to the FreeBSD Project. Talk >about clueless corporate idiots.

    The only clueless idiot around here is the gobshite who posted the prior article.

    >Ah, whatever. Sorry for the rant - this whole HA >scene seems to be more annoying than the rest of >the bunch. They all operate in "a customer who >needs that must have money to burn - let's >charge him hefty" mode.

    Hmmm you have a peculiar notion of what is HA. Banks etc are not interested in mickey mouse h0x3r solutions with no support.

    greg

  • Moderate this the "dumbass" guy down! What the hell? How did he get moderated up to a 2/

    Jerk.

  • In addition to linux-ha [linux-ha.org], which includes links to Linux Virtual Server [linuxvirtualserver.org], Piranha [redhat.com], Ultramonkey [sourceforge.net], you can also find organizations that do this for a living. One (the company I work for, to be honest) is Mission Critical Linux [missioncriticallinux.com]. Specify what your needs are, exactly (web service, database failover, file system, etc), then look around.


    By the way, is your consultant a reseller of Solaris (since I see he suggested that)?

    jeff

  • Looks like your consultant is shooting himself in the foot. There are many ways to produce failover-- a couple bash scripts could probably do it. I think your consultant just wants to make a sale of some of that expensive hardware, or is not creative enough to think of alternative methods.
  • DecNet is obsolete. Plain ol' ethernet bridging (not to be confused with routing) requires the ability to set arbitrary MAC source addresses on each outgoing packet. Various IPX-based failover and routing systems require this capability too. Also the Ethernet people don't always get their standards straight, requiring support for yet another header format.

    I haven't actually seen a network card that doesn't support arbitrary MAC, but I suppose that some old 8-bit ISA cards may still exist--if you have such a beast, mount it on your wall and get a network card capable of at least a megabit of throughput after host bus overhead.

    Note this change usually isn't permanent, i.e. we're not overwriting the NVRAM on the card or anything. The capability is simply due to the fact that the chip doesn't prepend the ethernet header for you, so the software has to fill in the second six bytes of each packet. Linux reads the MAC address from the card's ROM as a default, but you can override this with 'ifconfig', and for Linux bridging the source MAC address is set for each packet forwarded across the bridge.

    I'm willing to bet that a lot of NIC chipset designers intend for their chips (or at least most of the die) to be useful inside switches as well as inside network cards. Why design two different devices when you can just design one and sell it twice?
  • Right now I have on all my sun boxes, 2 fast ether cards, each on its own Cisco catalyst.
    Both hme0/hme1 have the Same IP and Same MAC address.
    Just a little perl script, failover the card and reconfigures EFS on the fly.
    Works ok, but could be more automated. This would work perfectly on any Linux box.

    Then of course there is multiple catalysts, local directors, multiple pipes with multiple carriers, OSPF... Very good network design.
    Only outage is when GTE/USWEST cuts those damn fiber cables....

    Right now we are working on 100% failover, 5 Nine HA solutions. Sun cluster, Veritias file system, Oracle HA solutions. Even moving up too a few Sun 10Ks.

    Depends on how far you want to go for Reliability.

    -IronWolve

  • The consultant says Linux is "shooting itself in the foot" for not supporting failover?

    I've got news for your consultant; Solaris folks buy a third-party product when they want failover capability, such as Legato (formerly Qualix) HA+.

    Is Sun shooting themselves in the foot, too?

    Third-party products are available for Linux, just like they are for Solaris etc. Buy them if you need them.

    --
  • Read what I said again; did I say "need", or did I say what we *DO*?

    Sun Cluster certainly exists. Legato HA+ does a booming business.

    There are reasons for that; but since you seem to think you're the expert, I'll leave their discovery as an exercise for your.
    --
  • Ok, I don't want this to sound like an ad, but I work for a company that has a product that should fit your requirements. I work at a startup (called Netboost) that was recently acquired by intel and our primary product is a strong arm based dual port NIC with Linux, BSDi, Solaris, and NT driver support. Here's a url [intel.com] with some info. I'm not in marketing, so I don't even know how you'd go about getting one of these (though you could send me mail and I could try and point you to the right person if you were really interested). There is an API that lets you write code for the NIC to handle packets as well as a bunch of sample code (some stuff I write). Anyway, check it out if your interested.
  • Rather than failover, consider using a load-balancing device. You'll get use of all of the boxes (and more can be added as traffic dictates) and it will automatically stop routing traffic to a downed box. It can be transparent or semi-tranparent to the servers. Of course, the application has to support being run from multiple servers simultaneously.

    There are a number of commercial products from F5, Alteon, Extreme, Intel, and others.

  • Slashdot has been saturated with 'IANAL' for several weeks now. I think we need a new acronym for this discussion: IDKCAHA. 'I Don't Know Crap About High Availability.'

    I've been trained in IBM's high availability product, HACMP [ibm.com], 'High Availability Cluster MultiProcessing' and manage a few production clusters.

    This is typical of what I'm hearing on Slashdot today...

    If a network node goes down, it's better if network equipment handles the failover.

    It ain't that simple. The node that went down has resources other than network ports. What about the application? What about the filesystems? What if the network connection is up but the application or filesystem is down? Show me a chunk of networking equipment that can handle those failures and I'll send you a dollar.

    In our production environment, we have an Oracle database running in an IBM HACMP cluster. The SSA (think: SCSI over token ring) drives are shared between the primary node and the backup node but can only be active on one node at a time.

    Should a network card fail, in the primary node, it will down the sick interface and bring up the IP and MAC on the backup card. Linux, too, can do that rather easily.

    But what if the problem isn't the network. What if the applicaiton on the primary node has failed? HACMP can down the primary node and bring the application up on the backup node, taking the disk drives with it. I have yet to find a Linux tool that will do disk failover.

    I've also seen comments in this discussion to the effect that secondary heartbeat paths are 'silly [slashdot.org]'. Obviously, the person who made that comment is insane. (We use a target mode SCSI over SSA for a heatbeat.)

    If your only heartbeat is running over the network and the network fails, neither node knows if the other is up and both nodes will attempt to claim the disk resources, come up on the same IP/MAC address and start the applicaiton. This is Very Bad. What do you think will happen when the network comes back online? I'll give you a hint. All of a sudden you've got duplicate nodes on the network. That's never good.

    To answer the consultant question, I think the person is stearing you in the wrong direction. Either you need to rethink your failover solution or you need to move to a platform that better supports the type of failover you want to do such as AIX or Solaris.

    Real Soon Now, I expect that there will a viable Linux HA solution. However, there ain't one now and that's where you are. Since you're a B2B startup with venture funds to burn, I suggest you throw some of that money toward IBM's HACMP or another commercial solution on a platform that has had an HA solution for more than a few months. I'd hate for your company to save a few bucks by using Linux and lose lots of money to downtime.

    InitZero

    (let the flames begin)

  • Yup, sure... append lines in lilo.conf are unformatted strings passed to the kernel. The kernel then passes them to the appropriate routine (or module, if you're modular) which does whatever the module-specific commands are telling it to do. In this case I'm telling Donald Becker's network drivers not to stop looking for 3com cards until all four have been found. For more details see the documentation.
    --Charlie
  • For those of us who do not yet know, what does "B2B" stand for? Thanks, -S

    Scott Ruttencutter
  • Free as in beer.. No
    Free as in non-proffit...
    Sun dosn't make any money but your paying for shipping.
    Thats not free...
  • An LD will probably do it but not at the machine end. If a NIC goes down it goes down.

    There's also a problem with the LD if you need sessions in your application. As long as you have big enough servers that can handle all the traffic coming from behind proxies you can always use the IP-sticky in the LD but experience with this has shown me that sometimes the load balancing can be really bad with this scheme. If the LD just happens to throw three or four big proxy addresses at one machine it will get bogged down and the whole idea with the LD is lost.

    There is ofcourse the ssl-sticky for ssl sessions over http. But there's a problem with that too. If the client uses IE and sits behind a proxy the ssl-session ID will not come from the client. IE by default uses http 1.0 when talking to a proxy which doesn't allow sessions. This is probably due to the fact that up to a very late point in time MS supplied proxies that talked only 1.0 and now they correct that by disabling the newer protocol in their browser.

    At the webserver end it seems like the client would be talking 1.1 but that is not true since the protocol info comes from the proxy, which talks 1.1.

    There might be a solution to this but I haven't been able to find anything yet. Otherwise the LD is a working solution that also provides failover for itself if you have two of them.

  • Hmmm...

    Did I say it was anyhing else? I only stated that I didn't use the Pirhana package.

    In fact, according to the LVS site, its not linux virtual server.

    Piranha is the clustering product from Red Hat Inc., it includes the LVS kernel code, a GUI-based cluster configuration tool and cluster monitoring tool.

  • If this is important enough to require high availability, and you are forking out the dough for all the cisco gear...

    what other software are you using, and why are you cheaping out to use linux?

    I mean, look.. I love linux... don't get me wrong. And linux *can* do this, but it'll take work.

    But if you want something that already does it, and considering the money being spent.. why not go with what the consultant said and pick up the suns he recomments and the failover gear he recommends?

    Sheesh. I wanna use linux at work too, but when it comes to a platform for a $60,000 piece of sofwtare, it sure didn't make much sense to argue that linux was 'cheaper' than solaris..
  • Because having an operating systems allows for custom tweeking. I have written numerous perl scripts to do VIP IP accounting and uptime statistics outside of the BigIP interfaces. Having a device like this adds FLexablility. As for the Local Director or ServerIron these platforms tie you to their interfaces to access data without the option to customize.
  • Those who can, do; those who can't, consult.

    funny, that's why I just got into consulting...

    I figured if people who don't know what they are doing can make good money telling people how to run their shops, think of what someone who does know something could make. :-)

  • As the other poster said, it depends on the driver. Thats the bad news

    The good news is DecNet requires the driver (and hardware) be able to change the MAC address. Thus even for cheap cards most of them can just in case the vender ever has the chance to sell to the last shop out there still running decnet.

  • Many cards can have their MAC address set. Linux ethernet drivers support that where available.

  • http://www.us.buy.com/comp/product.asp?sku=1016087 0

    This is D-Link's 4-port 10/100 NIC.. It has Linux drivers, and it's only $165.
  • So, in the telephony industry, this is a big requirement... The problem is that you don't want the outage of a single network link to take your machine out of service, or to interrupt any existing transactions.

    So, for example, if you have a TCP stream going to a specific NIC and the link between the NIC and the switch gets cut, or the NIC fails or something, then you need to be able to continue the same TCP stream on a second interface.

    You end up with several issues: On a lot of NICs, it's not that easy to figure out when it's having problems. Secondly, the second NIC is typically at a different hardware address, so you need to update the ARP cache of any machine sending to you. And, you have to figure out how to tell when the first NIC is working again.

  • I am wondering why NICs with more than one port are so danged expensive though? I can see a bit of an increase in price, but there is no way these things should be $400 and up (last time I looked..)

    The SBus QFE part may have been space constrained (SBus cards are small), which will bump the price a little. Multi-port PCI NICs normally need a PCI bridge part (actually it's been a while since I bought one, maybe they do it all in one multifunction PCI chip now), which pushes the cost up a little too.

    But the big reason is economy of scale. It costs a lot of money to design a product, document it, write drivers, set up distribution channels, and so on. Cost that is mostly fixed regardless of how few of the product you sell.

    Contimplate the following example:

    Assume for the sake of argument that it costs $1,000,000 to design a PCI board. Now assume I make a 4 port ethernet (with a parts cost of $40), and you make a one port ehternet (with a parts cost of $10). Also assume there are (only) 1,000,000 people on the earth (and all want to be in on the big LAN party). Some of them are uber-graks and will buy the 4-porter so they can have a 4-porter. Some want a "reliable gaming experance" and will buy the 4-porter because they have 3 more ports if hte first fails. Some want to run the LAN server and need more bandwidth. In all 100,000 people are intrested in my product. 900,000 in yours. To exactly cover our costs you need to charge $10 for the parts and a bit over $1 for the "overhead" -- a $11 price, I have to charge $40 and a bit over $10 in overhead -- a $50 price.

    Alot of the people who wanted a "reliable game experiance" are now swayed by your argument that they can buy two cards and get "enough" reliability. Or even 4 of yours ($44), and an extra $6 to buy another ethernet cable in case their breaks! A few more are swayed by the argument that $50 is alot to pay for a network card, look over there a $10 card. Maybe they should keep the rest of the money, or buy a new game, or save up for a monkey. Soon only 10,000 people want my card. Your overhead drops a little (it is still about $1), but mine rockets to $100!

    With a $140 price tag even the uber-geeks start rethinking, and decided maybe they would rater show their geekeness with a $130 EFF contribution, and a nifty EFF bumper sticker on the side of their case.

    That's when things really start to suck, only the 5 guys holding the LAN party that need my card are now intrested in it. The price rockets to $200,040. At that price the 5 guys will spend a long time trying to figure out a way to do the whole gig without my card. In the end maybe they just charge everyone on the planet $5 to get into the LAN party and end up with "free" cards.

    There are lots of little things wrong with this example (the guys running the part could probbably use 4 of your cards at once), there are more then 1,000,000 people, the overhead costs can vary from product to product, some people will buy even seriously overpriced goods. But I think it does go a long ways towards showing why a Sun QFE costs $1,500 and a Intel Ether Express 100+ is $25.

  • For a "top flight consultant" you have a few mis-conceptions. BSD isn't a new version of the Solaris code. Solaris is NOW a SysV derivative, i.e. ATT code. BSD is BSD. For the older versions of SunOS that you are thinking about, they separated from the main BSD tree ALONG time ago.

    As for offering your customers a product with a company who stands behind it's guarantee - you're giving them MS? Why? That is pure FUD. Did you hear about the court case that handed down a couple weeks ago where the software supplier was held immune due to the "we don't guarantee this software for any use" clause in the shrink-wrap agreement. Pretty much leaves the concept of a "Big company" being needed out in the cold.

  • Well from what I've gathered from the current discussion is somewhat of a lack of direction. So here are a few things to consider and answer before going forward:

    1) Do you need High Availability of 1 machine? (ie 99+% of a single machine) If the answer is yes, then clustering is the way to go. But doing that right is very expensive (hardware, software)

    2) Does it make sense to have a farm of identicaly configured machines? If you're using Linux / FreeBSD as your webservers and if you only run web servers on them, then you can get away from clustering proper and just throw a ton of machines at the problem. ie farm of web servers.

    3) Sounds like the Consultant has the right idea with the "expensive Cisco hardware" in making sure Layer 2 is fully redundant. Good step forward. Now ya just need to make sure your hardware that is connected to it will utilize it. Do you?

    4) If your running Solaris, then Alternate Pathing becomes your friend (especially with Quad Fast Ethernet cards), as well as Dynamic Reconfiguration. Are you, or is this a moot thread?

    5) Overall, what are you trying to accomplish? Uptime of hardware, uptime of the application, or raw uptime of the web servers? If you got a set up like /. then clustering proper is not done. If you can see, they just load balance 3 web servers, and then dedicate a box for ads, a box for the database, and a dedicated image box. And we know how little /. is down...

    Basically, that's pretty much it. Personally I wouldn't bother with clustering or complicating the web servers that much, I'd cluster the back end supporting stuff for the web farm. ie the back end database, fully redundant hardware, alternate paths and so on. And then let Cisco's Local Director take care of load balancing and checking the web server is up or not. (From what my network guy at work tells me, it can do that. I won't personally believe it until I see it).
  • The word used to be "supplier".

    Oh well at least I'm not seeing "architect" used as a verb anymore. I was just itching to shoot someone then.
  • I worked on a distributed systems project that needed high reliability LAN connections. The solution we used was a custom NIC that had two Ethernet interfaces and a 68000 with 512K of RAM on a single PCB. Each system broadcast heartbeat messages on all attached LANs. If a primary LAN failed or became partitioned, all systems automatically switched to the backup LAN. This was transparent to the processes sending and receiving data on the network since the NIC routed packets to the Ethernet interface designated as active by the system's LAN monitoring and failure detection software.

  • I could give you a detailed rant style answer but I think it is not worth it.

    Most root DNS servers, primary mail relays, etc use exactly what I said. And there is no such thing as what you said. Been there done, that.

    Please get a clue.

    Solutions using routing protocols cause serious trouble if and only if designed and ipmplemented by Minesweeper Consultants and Solitaire Experts.

  • if you only require service availability (but not session availability).

    I have to remind you - you do not use physicals. Apache listens on loopback only. So the client retransmits, it goes via the other interface and you have no problem. Session is alive.

    talk to (resp. across) a small set of routers (or routing protocol using hosts).

    Correct. You talk to two routers or just differnt ifaces on one that connect you to the backbone (via different layer 2 devices - switches or hubs). And from there on with the entire internet.

    In a similar internal corporate scenario you talk to the routers or the RSM on the switch that separate the servers from the lusers.

    I can give you a number of examples where it won't work at all.

    Yeah, sure. I have seen gazillion of b0rken network designs written by experts. Most of them with a minesweeper and/or solar sertificate. I am not beeing biased but core networking is not a subject in neither of these sertifications. Officially core network support in Slowarez is considered with a "to be or not to be" status in Sol 8. Check the zebra archive for details. With minesweepers it is not even considered.

    You don't happen to post in certain de newgroups ... ? This somehow sounds ... familiar

    No. Never used news. But I am not the only BOFH around.

  • Very good besides the fact that Layer2 failover has always been less reliable than layer3. If layer2 was better the internet core would not use OSPF and BGP.

    So, overall: OSPF instead.
  • Very good approach.

    The problem is that a bunch of carma w** who are out of their scope have immediately flooded the article with comments about piraniah, clusters and other irrelevant things. The question is about failover in case of link failure. The consulatnat thought of winhoze and chose layer 2. You have a unix system. Unix knows about routing and IP. Hence what you need is a layer 3 solution. For example:

    http://slashdot.org/com ments.pl?sid=00/05/21/1853216&cid=90 [slashdot.org]

  • Oh, that was a good joke. Yes, Nokia uses it. And nobody else.

    Juniper, 3Com, and Alcatel was at least working on it for a time in 1999. Yeah, that sounds like "just Nokia". :)

    There's also HSRP. Everybody and his dog uses it in their Cisco routers. Why not take that ?

    HSRP is a hacked version of VRRP v1. Where do you think they got the ideas from???

    And no, I don't work for the IPRG group. I've got some friends who used to, and one that still does, but no, I don't work for them.
    --

  • When the consultant installs a network that is clearly not designed for the needs of the company (i.e. supposedly requires special hardware and drivers that the consultant doesn't know how to integrate with your core product) you are being taken for a ride by people with little knowledge and less moral backbone.
    If you need multiple ethernet interfaces on a machine they should be separate cards for robust redundant failover. I run 12 linux boxes with 4 ethernet cards in each; my /etc/lilo.conf files look sort of like this:

    boot=/dev/sda
    map=/boot/map
    install=/boot/boot.b
    prompt
    timeout=50
    image=/boot/vmlinuz-2.2.5-15smp
    label=linux-smp
    append="ether=0,0,eth1 ether=0,0,eth2 ether=0,0,eth3"
    root=/dev/sda8
    initrd=/boot/initrd-2.2.5-15.img
    read-only

    The append line activates my additional ethernet cards, all of which are 3com 100bTs using Donald Becker's excellent open-source drivers.
    Combining this with round-robin DNS using the latest ISC BIND code, you can get incredible fault tolerance at a very low cost. You can even do IDE RAID (hard or soft) if you are too cheap for SCSI, and you can use rsync to keep your servers clones.
    Unless your application is extremely unusual and non-wwwebby, you can accomplish what you need without any expensive Cisco stuff or fancy double-headed cards at all. The consultant is taking you to the cleaners due to greed or a total lack of competence.
    --Charlie
  • For a big company name I'd have recomended SCO not Microsoft.
    HP is also good but my personal bise prevents me from recomending them for software solutions.
  • The Linux Virtual Server [linuxvirtualserver.org] Projects stuff might do the trick. I've used it for mailserver load balancing under medium load, and found it to be quite reliable, and easy to configure.

    Without RedHat's Pirhana package. ;-)

  • Well.. if you say "Here's what the consultant told us was the solution to our problem"..
    where's the solution? Was he just speaking theoretically?

    A dual port nic sounds strange, especially with this behavior. From a networking point of view, this makes sense.

    Sure, a dual port nic will help you, *if* it's set up to get arond transciever failure by bringing up the other port.

    Two nic's would be better, where the box itself could attempt to configure and use the other nic if it loses network connectivity.

    An even better (and more obvious?) solution is to have two computers..... complete redundancy.

  • Comment removed based on user account deletion

  • I have an alpha implementation of VRRP for Linux that I'll be GPLing within the next week or so.

    We're using it and it seems to work very well.

    Currently for 2.2.x only.

    Watch for announcements.

  • This is not something that's *in* any OS, unless Sun's added it into Solaris in S8. (Could be, I don't get to play with Suns anymore... sniff...)

    Although I'm sure the options have changed some since I was fully up on this stuff about three years ago, there were only a handful of failover options at that point, and only one of them worked really well.

    That one, interestingly was in reality a bag of (very good) scripts, which implemented a heartbeat function and when it detected something wrong, would down the interfaces, re-plumb them if necessary, reset addrs, and up them again. Although it's worth the money they charge, if you're into a serious DIY mode, there's no reason you couldn't write such scripts yourself, and there are almost certainly some already out there, probably as part of the Linux HA project.

    Oh, and as an aside, I would stick with the script-based solutions whether you build or buy: they're more reliable, and they leverage the OS better than the proprietary methods. (Qualix's main competitor back when I worked for Sun consulting for customers on such things was OpenVision HA, which was a huge, slick, impressive monolith of GUI binaries that had a well-earned reputation for leaving a trail of dead bodies behind it. FirstWatch, on the other hand, was simple and unimpressive in a demo, but it just worked, and worked well, in the real world.

    Qualix was bought by Veritas a few years ago - check with them if you want a decent supported package. (And let's face it - HA is certainly one area where it may not pay to roll your own, since a failure in the HA system in production would be a serious career-limiting move...)
  • My experience with consultants is that a good many of them are clueless. The reason they're consultants is they can easily BS the customer into believing they know what they're talking about long enough to bleed you dry...

    Interesting? This gets moderated as "Interesting?"

    It's flamebait, and if the moderators weren't so blind from anti-corporate propoganda...

    Let's see here...either (1) you've never worked in a corporate environment where you've had to deal with consultants or (2) you're a consultant yourself and "resemble that remark." From the (admittedly limited) experience I have with them, the original poster's remarks were on-target, though. Those who can, do; those who can't, consult.

    It's not an "anti-corporate" bias; it's an "anti-moron" bias. :-)

  • Our company uses these nifty machines for load balancing and fail-over. They are basicly x86 based machines running FreeBSD and some proprietary software. They also have the important things in life like 2 NICs and a nice rack mounted chasis. It is a bit pricy, but you get a very useful manual, support, someone to blame when one is on fire, etc. Most importantly, it works...

    One thing... Make sure you're pluging it into 120VAC. The power supply get's very unhappy if you don't... You learn these things when someone labels a 240VAC strip as 120...... Go figure.
  • To take that further, aren't they legally obliged to hold their shareholder's interests above those of their customers...

    I'm not sure, but if true, I find that prospect somewhat revolting. It's a basic admission that companies care more about money than about quality. Usually smaller companies are okay, but the big conglomerates make me skeptical of the good of capitalism in the big picture.

  • And if all our customers fuck off because we don't keep them happy, the shareholders are going to love that right? A satisfied customer is a paying customer.

    But you said, a few posts up, that your customers want a "tried and tested platform backed by a company that truly cares about their satisfaction." But now you imply that your company doesn't truly care about their satisfaction, but only about truly about their money. Which is it?

    I assume you care only enough about their satisfaction as it will bring in the dollars. Ie, you want to barely keep them satisfied enough, such that they'll buy more products. Such is capitalism at its extreme. You choose money over product quality.

  • No, the average corporate customer we get is more interested in a tried and tested platform backed by a company that truly cares about their satisfaction rather than being at the whims of the "open source" bearded hippy crew and their communist fuhrer.

    Hahaha, a sleezy capitalist fearing his/her eventual demise. Anyway, doesn't this 'company' you speak of truly care more about their shareholdrs than about it's customers' satisfaction?

  • > I just called up 3com and said, "Please send me
    > two of those pieces of hardware with

    Well thats nice. Look, I have no use for these
    things myself. I don't know what the product is
    called, I never bought one. I was simply trying
    to offer an idea and point in the right dircetion.
    I never claimed to be able to do more.

    I probably could find out the name of the product,
    but not in the time frame where it would matter
    wrt slashdot comments.

    > I think you want two servers with the same RAID
    > array....[snip]

    Yup...a very good way to do it...I agree (of
    course it doesn't handle the raid array itself
    having a catastrofic failure...but given the
    redundancy in a good array, that should be more
    rare than a system blowing)

    >> Of course, why thats even needed is beyond me.
    > apparantly..

    Thank you for changing the order of what I said
    so that it looks like I said something different
    than I did.

    If you were to look at my original comment, I said
    this about the case of SIMPLE ethernet line
    failover NOT the redundant servers case.

    -Steve
  • Yup, I know all that.

    However, you should note that I offered 2 solutions. One of them being almost exactly what he asked for, but implimented in hardware (and as someone else pointed out, possibly firmware too) which requires no driver software to work (beyond that of whatever existing ethernet card one has)

    The other solution, yes its alot more costly. Yes it MAY not be right for the given situation. However, I felt it should be offered up anyway, and to let the person in that situation decide.
  • I am wondering why NICs with more than one port are so danged expensive though? I can see a bit of an increase in price, but there is no way these things should be $400 and up (last time I looked..)

    Sorry to be a bit off-topic, but there is a reasonably priced 4-port ethernet solution out there. Compex, Inc. [cpx.com] makes a quad port ethernet card (P/N FL400TX/PCI) that sells for $189.95 on buy.com. Looks like it's out of stock right now though.

    I purchased one of these for our server (Linux based of course) here at work and have been quite happy with it. I'm using it for subnetting our network (vs. fail-over network links.)

  • B2B == We sell to businesses. Why can't they just say it?
  • Your comment is very informative, however, what he's really talking about doing is implementing Cisco Catalyst switches that use HSRP (hot-standby routing protocol) and load-balancing in order to give you twice the throughput, without using two separate subnets. This is the preferred and desirable way to implement high availability and layer 2 redundancy. It can be done on both Solaris and NT (don't know about linux). The point most people are missing here is that it is preferable to do this in hardware, as opposed to software, because the hardware tends to be more reliable. I would trust Cisco IOS to handle my redundancy much more than even Unix (although Unix is very stable). I think most people are answering the incorrect question. Now that he has all of this hardware, how does he use it? I would be interested in hearing if there are any devices or device drivers that allow you to do this in Linux.
  • This tactic works...and a version of it has been used in commercial minicomputer systems for a while. I haven't been to the linux high-availabilty site yet, but they probably got the idea from existing commercial products.

    I worked on an HP-mini, and it used a similar setup. Basically, the two *identical* minis shared a SCSI bus with redundent media. The backup mini would ping the other one over the SCSI bus, and if it didn't get a response it would take the IP of the first one. Worked damn well.

    The only drawback is that the backup isn't doing anything but issuing a ping, mirroring the system RAM in machine 1, and waiting. The upside is that short of a missle strike, you had very high reliablity. Most failures didn't even cause a pause.

    I don't see any problem with using the same method with more systems, though a cluster starts to look attractive after a while.

  • I recently attended the SGI University, and besides recieving a little Tux doll, I sat through some fairly inane discussions about SGI [sgi.com]. BUT, this may help you out... they had a session on clustering for high availability. They mainly spoke about being able to hide planned downtime by using 'High Availability, Mission Critical' Clusters where if one failed (or was shutdown for maintenance) the other would automatically pick up the slack... they also spoke of this with regards to Beowulf clusters... you may want to check out their open source area [sgi.com] also, and see if any of this software is publicly available yet. It does sound like they are heading in the right direction with getting into Linux...
    Hope this helps
    regards,
    Benjamin Carlson
  • Every Ethernet chip you are likely to encounter allows temporarily overriding the MAC address in software. Almost every Linux Ethernet driver (all the ones written by me) allows changing the MAC address while the interface is down. The driver rewrites the chip's idea of the MAC address when the interface is brought back up.

    This feature isn't needed for all fail-over schemes, but it does exist for those schemes which use it.

  • Sounds like a Cisco LocalDirector or one of it's competitors could do the trick

  • Solaris "Alternate Pathing" (Score:3)

    by Anonymous Coward on Wednesday May 24, 2000 @05:04AM (#1051243)
    The Solaris feature refered to is AP or Alternate Pathing. It is used for disk devices and network devices. Basically, you create a psuedo network device. This psuedo network device is attached to two real network devices. Since the OS is doing all network activity through the psuedo device, it is capable of working out of either real physical network device without the applications (or the rest of the OS) being aware there was a change. It is similar for disk.

    The main reason for AP is for the DR or Dynamic Reconfiguration feature. If you've got three system boards, then you can have some redundant hardware so that you can take down and remove a system board *while the OS is still running*, and keep your network connection going without missing a beat. (Same for disk.) Neat stuff.

  • Been there, wrote it myself (Score:3)

    by bluGill ( 862 ) on Wednesday May 24, 2000 @05:01AM (#1051244)

    I wrote software for Solaris (Which as others have pointed out does not do this without 3rd party software) because we found that no solution would fit our needs well. When looking at the prive tag we concluded that we could do better. (Come to think of it there is High avaiable Solaris, but it isn't cheaper or better then 3rd party stuff)

    Basicly we ping something on the other side of the router every 5 seconds, and if the ping doesn't come back we switch to the other port. That is the overview, but you need to do some more isolation before you blindly switch ports.

    I strongly recomend you put in some other path between you and the box you are pinging. Several times we have been bitten when the box we weere pinging went down and not the router, or alternaticly the network was so busy the ping didn't get through within our timeout period.

    There is no portable way in solaris to tell if one ethernet port has signal. You can find out from some drivers, but when you change to a different ehternet card you have to do something else to find out.

  • linux option (Score:3)

    by mattdm ( 1931 ) on Wednesday May 24, 2000 @04:47AM (#1051245) Homepage
    http://www.linuxvirtualserver.org/ [linuxvirtualserver.org]

    including

    Piranha [redhat.com]

    --

  • dual port nics? (Score:3)

    by kevin lyda ( 4803 ) on Wednesday May 24, 2000 @08:20AM (#1051246) Homepage
    and when the dual port nic dies and takes both interfaces with it you switch to psychic networking? i'd really suggest two nics not a single dual one.

    that said, linux can do routing. why not set up a loopback device and then have it route through either nic? ip was designed to deal with multiple routes, why must your consultant reinvent the wheel? (loopback addresses are published, so they'll be seen on the network)

  • Q: is this a good solution? (Score:3)

    by Felinoid ( 16872 ) on Wednesday May 24, 2000 @05:04AM (#1051247) Homepage Journal
    I hate saying "I hate to tell you this but" when I really enjoy telling someone something.

    So I won't lie.. I'm happy to tell you....

    This may not be the right solution.
    The problem isn't the consultents. They know there stuff otherwise you wouldn't be paying them.
    They come with years of experence and bieses.

    Expect a consultent who isn't friendly with Linux to dig up a solution that will not work on Linux.
    Yes Solarus can do a lot of great things Linux can not. In the end Linux has one great advantage and thats price. Source code and quick security patch relases is a bonus.

    This rule holds for an NT shop....
    And don't put a consultent past finding a feature Solarus dosn't have. Your talking with some of the greater frelance tallent in some cases and if a defect is to be found they can find it. Once found that defect becomes a case for switching to something the consultent likes.

    So once you pick a platform for your shop pick a consultent who is buddy with your choice. If then he recomends something else it will be after bleeding dry all posablitys.

    This may not be the only way to do it...

    and... it may be the worst way to get it done...
    Linux has it's limits don't get me wrong but you can allways find more than one solution. Linux may support 4 out of 8 solutions.. if your only presented 1... there may be a reason...

  • OS Level failover (Score:3)

    by PenguinX ( 18932 ) on Wednesday May 24, 2000 @05:37AM (#1051248) Homepage
    One of the things that I have found is that OS level failover doesn't always work or will have odd problems. If you are looking for Enterprise level uptime then hobbling together a solution such as this is not for you. The company I work for uses a cisco localdirector [cisco.com] to do the work for it. What's great about this sort of solution is that a localdirector will round robin, do failover, and such a dizzying array of things that it's wonderous. I would suggest you look into this solution or one similar

  • This is the wrong question (Score:3)

    by FascDot Killed My Pr ( 24021 ) on Wednesday May 24, 2000 @04:37AM (#1051249)
    Your question: "Here's what the consultant told us is the solution to our problem. Where can we get the hardware?"

    What you want to know: "Here's our problem. Here's the solution the consultant came up with. What improvements can /. suggest?"

    For instance, why do you need dual-port NICs? If it's just for the throughput, why not just use 2 single-ports? This also provides redundancy in the hardware department.
    --
    Have Exchange users? Want to run Linux? Can't afford OpenMail?

  • Re:Failure to implement open standards. (Score:3)

    by yesod ( 25715 ) on Wednesday May 24, 2000 @06:28AM (#1051250)

    I believe Juniper and 3Com also support the use of VRRP.

    You may not be able to implement HSRP without paying Cisco a license fee. I'm not sure if anyone has approached Cisco from an open-source viewpoint though.

    As for a public implementation - I should have a Linux VRRP implementation out this week.

  • network failure should be handled by network HW (Score:3)

    by jalex ( 93439 ) on Wednesday May 24, 2000 @06:15AM (#1051251)
    If a network node goes down, it's better if network equipment handles the failover. The server should only handle the failover if the network hardware is incapable of doing so. The server is the weakest part of the network, so you want as little as possible depending on the server.

    Say you have a machine with two dual port NICs or even two NICs. Have a script that checks the main network interface every ten seconds. If the main interface becomes unavailable, unload it and load the second interface with the same IP and reset all of your routing information.

    If you have a server with that kind of "need" then maybe you should consider having a better routing setup altogether. Consider how www.netscape.com will actually resolve to several IP addresses. The options are numerous. The main issue, is that linux works just fine. (Although freebsd has sexier networking)

  • Hardware Support (Score:3)

    by TheCarp ( 96830 ) <sjc AT carpanet DOT net> on Wednesday May 24, 2000 @04:41AM (#1051252) Homepage
    > We are a growing B2B company;

    Good to know that you are buzzword compliant...
    I understand thats very important to some people,
    and if I ever figure out who those people are, I
    will probably avoid them like the plague.

    As for fallover...check out 3com....long ago a
    man (who would later go on to teach Unix courses
    at WPI and be one of the best teachers I ever
    had for anything) designed a piece of hardware
    with 1 ethernet port on one side, and 2 on the
    other...it was designed to do JUST THAT.

    Completely in hardware. He did it for a company
    that was later bought out by 3com...he claimed
    (a couple of years ago, when I was in his course)
    that they still sell the product that he designed.

    Of course, why thats even needed is beyond me.
    For better redundancy, you really want seprate
    redundant servers, each with RAID arrays and
    probably a couple of localdirectors (or round
    robin DNS for a cheaper solution) direcing
    connections between them (giving both fallover and
    increased availability) but...thats just IMNSHO.

    Afterall, if a CPU fries, or a power supply starts
    letting its magic smoke out...all the duel port
    NICs in the world wont help.

  • Other thoughts (Score:3)

    by Ron Harwood ( 136613 ) <(harwoodr) (at) (linux.ca)> on Wednesday May 24, 2000 @04:53AM (#1051253) Homepage Journal
    Making the assumption that you want a web-farm, and for maximum availability, you are using two cisco switches, and two cisco PIXes, and two cisco local directors... you can still get away with a single NIC (not that you have to) put half of your web-servers on one switch, and half on the other.

    Otherwise, you can put two NICs in (one on each switch) and assign them each their own IP address... no need to fail over... although I would look at the F5 BIG/IP - as it can make sure that your servers are serving up content... the Local director isn't as good at this.

  • redhat piranha? (Score:3)

    by mikeee ( 137160 ) on Wednesday May 24, 2000 @04:53AM (#1051254)
    I think piranha does just this. Byte just ran a look at Linux HA clusters: http://www.byte.com/column/BYT20000510S 0001 [byte.com]

  • Re:huh ? (Score:3)

    by dwakeman ( 171962 ) on Wednesday May 24, 2000 @04:55AM (#1051255) Homepage
    http://support.in tel.com/support/network/adapter/pro100/30504.htm [intel.com] That is the dual port adapter that comes in Dell Servers....Has anybody used this before? D

  • Building Linux Clusters - O'Reilly (Score:3)

    by howard_wwtg ( 175831 ) on Wednesday May 24, 2000 @04:59AM (#1051256) Homepage
    There are many good ways (and even more bad ways) to build a redundant/failover environment. Your consultant is just "seeing the one solution he knows). As others have posted there are many good solutions already out there.

    Building Linux Clusters [oreilly.com] is just what you should read.... Uncoftunately it won't be out until August.

  • What you need (Score:4)

    by Anonymous Coward on Wednesday May 24, 2000 @04:47AM (#1051257)

    Everything you need is at High Availability Linux [linux-ha.org].

    I too am/have built a B2B exchange on the linux platform and found JServ [apache.org] to be *INCREDIABLE* at HA/Failover safe features.

    As for the 2 network cards for each machine, that too is a *VERY GOOD* thing. It allows you to partition out your network traffic to achieve much better response time. For example our network has 2 NICs in each machine. There is "Web Server to Database" network, There is a firewall to webserver network, and we have a seperate network for office web surfing and misc stuff like that. Access to the "WebServer to Firewall" network is handled across the router.

    One thing to keep in mind when dealing with DB aware web applications is that unless your code is *VERY POORLY* written the biggest bottleneck will be in network latentcy.

    -Grimsaado

  • Ignorance of options is not a failure (Score:4)

    by ch-chuck ( 9622 ) on Wednesday May 24, 2000 @05:10AM (#1051258) Homepage
    When Linux was cranking up last year the folks at TurboLinux [turbolinux.com] sales called up promoting their fault tolerant cluster solution, altho it's not free you could get a timed demo - so Linux solutions exist.

    Just a general observation - Linux is pretty well fleshed out with about anything you can think of in one form or another, it just isn't chasing you down with in-your-face ads and high pressure sales promos like other comercial products, so it may appear to be deficient but more often than not just a few days (for us slow pokes) search and trials will usually turn up an inexpensive quality solution in some stage of development hidden somewhere.

  • Failure to implement open standards. (Score:4)

    by jcostom ( 14735 ) on Wednesday May 24, 2000 @05:37AM (#1051259) Homepage
    It never ceases to amaze me. Companies want to sell you obnoxious amounts of software and hardware to do something as simple as create a highly available system. Not to mention projects like the Linux-HA people, who while are doing a good thing, are (IMHO) heading in the wrong direction (using RS-232 heartbeats is silly).

    Has anyone considered VRRP [ietf.org] (Virtual Router Redundancy Protocol)? It's an actual open standard, and it works. It not only works, it works amazingly well.

    One of the major users of VRRP technology is Nokia. They've done extensive work on the protocol, and use it in their line of firewalls (which btw run a heavily modified FreeBSD codebase).

    VRRP uses multicast packets that are similar to OSPF "Hello" packets to let the partner(s) know it is alive. If the primary machine dies, the backup instantly takes over. When the takeover happens, it not only assumes the IP address of the dead machine, but it also answers for the MAC address of the dead machine.
    --

  • Re:Linux High Availability project (Score:4)

    by SEWilco ( 27983 ) on Wednesday May 24, 2000 @05:13AM (#1051260) Journal
    Notice there are many links to related HA items there at the Linux High Availability Project [linux-ha.org]. It sounds as if you're looking for something like FAKE [vergenet.net], which lets a machine acquire the IP of another machine in a failure (note that FAKE points out that it has been moved into the "Heartbeat [linux-ha.org]" code at Linux-HA) -- although some link chasing is necessary to learn where it went.

    "...dual-port NICs...switch the ports when the active port fails...

    Oh, I see. When one port (or its path) fails, you want to switch the IP to a different port? I don't think "the driver" needs to do that, just change the IP assignments with ifconfig.

    • Monitor each link with some sort of heartbeat.
    • When there is no response, assign the IP of that link to the backup interface. Just use ifconfig to alter the interface configuration.
    • Have the backup interface be on the other NIC, not "switch ports" as you mentioned.
    • Dual-port NICs are not needed, if you can fit 3-4 NICs in your machine.
    • Have heartbeats running on backup and downed interfaces also, to report problems and repairs.

  • Re:This is the wrong question (Score:4)

    by x0 ( 32926 ) on Wednesday May 24, 2000 @05:37AM (#1051261) Homepage
    why do you need dual-port NICs?

    On Suns at least, the dual (well, quad) port NICs are used as a heartbeat signal between the active server and the failover box (when using FirstWatch).

    True, you could use two separate NICs in each box to provide the same solution, but then you are using up three PCI slots since the heartbeat NICs do not carry any packets.

    I am wondering why NICs with more than one port are so danged expensive though? I can see a bit of an increase in price, but there is no way these things should be $400 and up (last time I looked..)


  • Feh! (Score:4)

    by Greyfox ( 87712 ) on Wednesday May 24, 2000 @05:36AM (#1051262) Homepage Journal
    My experience with consultants is that a good many of them are clueless. The reason they're consultants is they can easily BS the customer into believing they know what they're talking about long enough to bleed you dry. They may even provide you an actual solution that may even kind of work but which is patently the worst way to do what you were wanting to do. Then when you DO get someone in who knows what he's doing, that guy will have to spend twice as long beating your company into shape because he has to go back and undo everything the previous one did.
  • What kind of application are you trying to fail-over? A database? A web-server?

    If you wanted a web-farm, that's dead stupid easy. Fail-over database/ftp/nfs isn't too hard, but (presently) requires commercial software. Understudy Polyserve [polyserve.com], Wizard Watchdog [wizard.de], or even RSF-1 [high-availability.com] are just some of the HA clustering products available.

  • Easy solutions (Score:5)

    by sdw ( 6809 ) <`sdw' `at' `lig.net'> on Wednesday May 24, 2000 @04:51AM (#1051264) Homepage
    First, it wouldn't be that difficult to modify the kernel to support this. I've released patches to work around bugs in the ARP reply with two NIC's. Routing to the active port shouldn't be too difficult.

    A non-kernel invasive version of this would be a script that configures one port with the desired IP/mask and creates the default route. It then puts the other port in promiscuous mode and monitors it for traffic using a libpcap based program or even a possibly modified tcpdump. As soon as it sees any traffic, it switches the configuration and starts monitoring the other port. This could probably be written in 2-4 hours given a network to test it in.

    For a possibly simpler solution (i.e. no code to write), use a pair of additional Linux systems. Configure each of them to load balance with LinuxVirtualServer (aka LinuxDirector) or the Pirhana version of it to as many backend servers as you have, BUT to Different internal IP addresses. Good choices would be 10.* addresses, say 10.0.1.* and 10.0.2.*. Using either a dual NIC or two NIC cards in each server, create two networks with one for each of the load distribution servers. Configure Apache et al to respond to the IP addresses of each network the same.

    BTW, there are 4 port 100-base-T cards out there, from Adaptec I think.

    Good Luck!

  • Linux High Availability project (Score:5)

    by dashuhn ( 12024 ) on Wednesday May 24, 2000 @04:40AM (#1051265) Homepage
    You may want to check out http://linux-ha.org/.
    The "heartbeat" application implements node-to-node monitoring over a serial line and UDP and can initiate IP address takeover based on a notion of resources provided by nodes and resource groups. It worked well for me. However, this was only a very basic two-node setup.

  • Shoot your consulatnt (Score:5)

    by arivanov ( 12034 ) on Wednesday May 24, 2000 @05:48AM (#1051266) Homepage
    Shoot your consultant. With a big gun. Only an idiot will suggest a layer 2 failover for a unix system.

    1. Your consultant should learn routing protocols

    2. Your consultant should learn the concept of a loopback alias.

    3. Your consulatnt should have an IQ of above 25

    4. There is absolutely no need for link layer 2 failover where layer 3 will do. Unix is not WinHoze. It knows about routing.

    So your task list is:

    • Configure loopback aliases on the linux boxes.
    • Configure apache to listen only the loopback alias interface.
    • Build gated from rhat sources they have the patches for linux-2.2 in already. You may use zebra CVS instead but it is still a bit off in terms of stability. You may need a script that HUPs it a few times gated as gated does not always start clean and update the routing table on 2.2.x.
    • Configure ospf on gated and on your cisco gear. Distribute default into OSPF as gated from the 3.5.x tree has no IRDP.
    • Shoot your consultant
    In btw: your bill is 500$.

  • Comment removed (Score:5)

    by account_deleted ( 4530225 ) on Wednesday May 24, 2000 @04:56AM (#1051267)
    Comment removed based on user account deletion

  • Wrong failure being addressed (Score:5)

    by igaborf ( 69869 ) on Wednesday May 24, 2000 @05:38AM (#1051268)
    You're attacking the wrong problem. What you need first is not Linux failover but consultant failover: Your consultant has failed; you need to switch to a new one instantly.

Slashdot Top Deals

"Religion is something left over from the infancy of our intelligence, it will fade away as we adopt reason and science as our guidelines." -- Bertrand Russell

Close