Mixed Reef - Failure as a Design Consideration
Posted by Chad , in Reef setup 19 January 2012 · 0 views
We have all heard the stories. We at WAMAS Waves have told them in our last three issues of the Marine Disasters series. Tank catastrophes caused by a myriad of problems that led to the loss of thousands of dollars livestock, damage to equipment, damage to our homes, hits to our pride, and hasty exits from the hobby. It is a risk that we accept in order to have these serene creatures, vivid colors, and part of the ocean in our homes. Right? Wrong.Errors are a part of every human endeavor. We all make them every day. However, when you anticipate the consequences of those errors and make considerations regarding the outcome, you can greatly reduce or eliminate problems resulting from those errors and associated heartache. This quarter in the Mixed Reef, I am going to share my system design philosophy that has been forged by 23+ years of marine experience and tempered by my work as a nuclear safety and reliability engineer. This philosophy is a concept that I like to call the "dual failure reliability criterion." What does that mean? It means the system is protected after any one failure and it would take two failures before a problem becomes an issue.Protecting your reef and all you have put into it requires a little time, effort, and sometimes money. But with all of the time, effort, and sometimes a lot of money you have put into it. The expense is worthwhile.Now, how do you go about applying the dual failure reliability criterion to your reef? By asking questions: "What can fail?" "How can it fail?" "What happens if it fails?" "How likely is it to fail?" "What can prevent its failure?" and "What can prevent or mitigate the consequences of its failure?" Do this for each part you add to your system and you will surely identify things you can do that will make big improvements to your system resilience.With that, let's talk through a few examples.Starting simple: What can fail? The heater. When asking "What can fail?" the answer is always an object, rather than an action. Thinking about glass breaking, seals leaking, or overflows clogging focuses your actions more than addressing the end result of water on the floor.How can it fail? On. Off. Leaches chemicals into the water. Adds a voltage to the water. Start by avoiding the cause of the failure (case breaks and then leaches chemicals into the water) and focus on the failure itself. Once you are satisfied that you have identified how the part can fail, add paths to reach that failure. On – sticky relays within the heater. Off – sticky relays within the heater. Leaches chemicals into the water – infiltration of water into the heater. Adds voltage to the water – infiltration of water into the heater. Notice that failure pathways can (and often are) the same.What happens if it fails? Answer this question more on a system level. For example, if a heater fails On and temperature rises too high, it can have adverse effects on the health of our animals. If a heater fails off, temperature may fall too low and have adverse effects on the health of our animals. Sometimes, the consequence of the failure is miniscule and not worth addressing further. For example, in my seahorse setup, the temperature was kept below 72 degrees and was rarely on. In that setup, what happens when the heater fails Off? Nothing that is worth the time or effort of the work to prevent the consequences. How likely is it to fail? Use your experience and the experience of others to determine the likelihood of failure. I use general time-related criteria here to dump the likelihood into one of a few different categories: It is likely to fail within a year, within 5 years, or not likely to fail over the life of the tank. Also be honest with yourself about your own maintenance habits. Many things are more likely to fail if they don't have routine maintenance performed. The obvious answer is to do the maintenance, but if you know you are likely to put off the maintenance and failure would result in a major problem, then you should put a failsafe in place. Based on my own experience and the stories of many others, I categorize heaters as likely to fail within 5 years. What can prevent its failure? This question will have at least as many answers as there are failure modes. And again, use your experience and the experience of others to identify solutions within our control as aquarists. To prevent water infiltration into the heater, the adjustment mechanisms can be mounted outside the water. Don't forget to add "do not use" to the list, as it may sometimes be the best option. For years, I did not use a heater on my aquarium. I noticed that they were hardly ever on in my house and decided to see what would happen if I just removed it. It worked in that setup, but this obviously is an option that needs careful consideration. Sometimes, the answers are beyond our control as aquarists. For example, we identified sticky relays within the heater as a cause for the heater failing both On and Off. The best bet on preventing failures from manufacturing problems outside our control is to purchase higher quality equipment with a good reputation and go from there. What can prevent or mitigate the consequences of failure? For the times when preventing a failure is outside our control as aquarists, we must resort to other means to protect our reef. In these cases we have two goals first can we do something that prevents the problem? And second, can we do something to make the problem more visible to us. For a heater that fails On or Off, the consequences can be prevented by connecting the heater through separate temperature controlled outlet to remove power to the heater if it were to fail On. For water infiltration into the heater that adds voltage to the tank, a grounding probe can be installed. In some cases, the consequence is not something that can be avoided; for example, water and electricity do not mix. In these cases we take deliberate efforts to mitigate the consequences and alert us to the condition prior to the problem spiraling out of control. To summarize actions for a failed heater, an aquarist can significantly reduce the likelihood of a tank problem caused by a failed heater by 1) purchasing a high quality heater with a good reputation amongst other aquarists, 2) installing the heater in a vertical orientation with the temperature control mechanisms out of the water, 3) installing a grounding probe, and 4) employing the use of a separate temperature controller in addition to the heater.Slightly More Complicated:For the next example, I am going to use Tom's (Origami) auto top off (ATO) system - slightly modified - for my discussion. In this system, it was Tom's desire to increase automation by connecting the fresh water supply directly to the ATO system. We all know that this is a recipe for flooding your home and adding too much fresh water to your reef and throwing salinity out of whack, right? Well, by looking carefully at what can fail and implementing features to prevent or mitigate the failure, the system can be automated.Starting with the minimum necessary components, a sump, top-off control and RO/DI System:Using the same process as before, we ask a few simple questions.What can fail? The solenoid valve and top-off control.How can it fail? The solenoid valve can fail shut or fail open. The top-off control can fail to engage or in the ON position.What happens if it fails? Either if the solenoid valve fails shut or the top-off control fails to engage the ATO system will not work as planned. In either of these cases the ATO system fails to add water to the system. If either the solenoid valve fails open or the top-off control fails in the ON position fresh water is continually added to your reef which can cause floods and lowered salinity.How likely is it to fail? The general conglomerate experience says float valves fail frequently and solenoid valves less so; however, failures still occur. Also, float switches tend to fail more frequently than pressure or optical switches.What can prevent its failure? Maintenance to ensure the solenoid valve and top-off controls are working can help. However, the failures we have discussed are largely manufacturing related. As such, the purchase of high quality components with a good track record is key. With the solenoid valve, always purchase a valve that fails shut. And as always, we can choose to not use the components.What can prevent or mitigate the consequences of failure? The two consequences we identified are floods and low salinity. To mitigate the flooding, we can install a drain on the sump; however, a drain does not mitigate the effects of low salinity. To mitigate low salinity, there are a few things we can do: 1) install a second valve and control mechanism, 2) limit amount of water that can be added to the sump to an inconsequential volume. In Tom's case, he chose to limit the amount of water that can be added and did so in two ways: adding a timer that would only allow the solenoid to be energized for a short amount of time in a day and by adding a ten gallon container. The ten container adds another component to the system that requires us to think through the same questions: "What can fail?" "What happens if it fails?" "How likely is it to fail?" "What can prevent its failure?" "What can prevent or mitigate the consequences of its failure?" Doing so identifies the potential for flooding in the ten gallon container, which Tom mitigated with the addition of a low volume pump such as a maxi-jet or peristaltic pump and the same timer scheme used before:Going through another round of our questions on the RO/DI system, Tom identified that frequent stopping/starting of the RO/DI system was shortening the life of the DI resin. To mitigate this, Tom added another larger reservoir for RO/DI water that he only fills when it empties and the associated control hardware and pump. And so we have the final system that is failure resistant and automated.To further increase the failure resistance of the system, Tom could add an overflow from the ten gallon top-off reservoir back to the 44 gallon reservoir and from the 44 gallon reservoir to a house drain line.Conclusion:As you can see, one of the drawbacks of this method can be additional equipment and associated expense. And of course, with additional equipment means higher complexity, more maintenance requirements, and more things that can fail. However, as long as you go through this same process, make failure-limiting changes and changes that make failures more obvious, it significantly limits problems and losses that may result from the broken equipment; thus warranting the increase in complexity.By asking a few simple questions, being critical of our systems, components and ourselves, we can greatly improve our reef's resilience to failures. Doing so will certainly require some thought, will result in changes to how you have things set up, and may result in additional equipment. However, when our reef is more resilient to failures, we can sustain fewer losses and enjoy the hobby that much more!