Fault tolerant, hot swappable subsea control architecture can improve reliability

SIL 1 can usually be achieved using standard hardware augmented with some combination of hardware and software diagnostics. To achieve a SIL 2 rating or higher often requires hardware redundancy, along with enhanced diagnostics that provide detailed information on system health.

Topside applications

Topside system equivalents that meet SIL 3 requirements have been widely deployed in the petrochemical, refining, and oil and gas production industries since the publication of IEC 61508. The most successful solutions developed in the 1990s were based on triplicate hardware platforms. Triplication of CPU and I/O allowed sophisticated voting schemes that provide greater fault coverage and diagnostics. This voting approach in a triplicated system is called 2oo3 (2 out of 3). A triplicate voting architecture provides a fault-tolerant capability that allows continued operation of the safety function, even though faults are present in the system.

Triplicated systems support hot swapping of failed components without shutting the system down or degrading performance. This is referred to as triple modular redundant (TMR).

An input from the field proceeds along three independent paths through the TMR system. The inputs are presented to each of the three logic solvers. The logic solvers vote on the validity of inputs and perform whatever logic is required by the application. Results of the three logic solvers then are sent to the output modules and each output module votes the results. The voted output is then sent to the final field device. At any point in the chain, any disagreement is resolved by voting and any errant module's output is not acted upon. In other words, two out of three signal paths must agree on a course of action.

Simplex SIL2 1oo1D architecture.

TMR hardware triples the number of components, increasing system size and cost. Often, that increased cost can be justified solely by providing the ability to repair the system online, allowing a critical process to safely remain in service. Any process that has a high cost associated with an unscheduled shutdown is a potential TMR application.

Simplex and dual systems

While TMR systems provide significant operational advantages, their cost and size make them difficult to justify for lower SIL applications. With advances in hardware and diagnostic software, higher SIL compliance without triplication has become possible. Even a simplex system can achieve SIL2 coverage if it has additional diagnostics coverage coupled with a second method to insure a safe shutdown.

The simplex 1oo1D (1 out of 1 with diagnostics) system meets the requirements for SIL2, because the diagnostics can detect a hardware fault and initiate a shutdown to insure fail-safe operation.

A 1oo1D system cannot continue operating in the presence of a diagnosed fault, so it must initiate a shutdown. Obviously a 1oo1D system does not support hot swap of failed modules.

Because the logic solver drives the diagnostics, a 1oo1D system cannot have complete diagnostic coverage. The addition of a second processor improves diagnostic coverage to the point that a simplex I/O with dual processors is capable of providing SIL3 fail-safe coverage.

Failure of any component in a SIL3 fail-safe system necessitates a shutdown. Failure of one processor would degrade this system to a 1oo1D, SIL2. If the process can operate at SIL2 for a short time, a failed processor could be hot swapped to restore SIL3 fail-safe coverage. Any failure of input or output modules would still require a shutdown.

By adding a redundant I/O to the design, a fully fault-tolerant SIL3 system can be achieved. While not as robust as a TMR system, this approach has the benefit of graceful degradation on module faults and supports hot swapping.

This architecture is widely accepted in the topside process industries and is available from many manufacturers.

Subsea control developments

Earlier subsea production control and safety systems were mounted primarily topside, with communication links to I/O installed subsea. The nature of these installations has evolved as more processes have moved underwater. Gas/liquid separation, pumping, and subsea compression require high-speed response, so closed-loop control cannot be accomplished with the communication delays inherent with topside mounting. This need for speed has helped delay the use of subsea applications requiring these features. A number of issues must be resolved before a more autonomous subsea control and safety system can be deployed.

Adapting topside technology

All the architectures presented can provide SIL2 or SIL3 coverage. The challenge is to adapt topside technology for subsea use. Subsea control and safety systems are housed in SEMs, which maintain a nitrogen atmosphere for the electronic components. Space is at a premium in SEMs and electrical wiring penetrations to the module are expensive. The redundant field devices required to achieve a certain SIL rating doubles the number of wires and vessel penetrations.

Redundant SIL3 1oo2D fault-tolerant architecture.

It is also far easier to environmentally harden and miniaturize topside equipment than it is to design and build new subsea enclosures. Most topside systems are mounted in standard cabinets, usually 19-in. (48¼ cm) wide. Box depths vary. Meanwhile, a SEM requires ROV installation and removal hardware and procedures. Topside installations do not suffer from the same vibration and environmental extremes as the subsea equipment, but they are critical factors for subsea systems.

To fit SIL-rated topside systems into existing subsea enclosures requires a redesign of topside hardware into a Eurocard form. Eurocard is a standard size electronic circuit board supported by all current SEM designs. It can meet the challenge of subsea requirements.

The reduced Eurocard size makes it difficult to find space for hardware triplication, eliminating TMR architectures from consideration. Dual systems, while smaller, still are too large to deploy in a Eurocard SEM. The lowest level SIL3 architecture possible is the redundant CPU SIL3 fail-safe. This architecture requires only one additional module over a simplex design, so the additional space necessary to achieve SIL3 is minimal.

The requirement for redundant field devices poses another problem for a subsea SIL3 system. Doubling I/O increases the size of the I/O modules and doubles the size of the wiring bundles. An obvious solution: Use two SEM enclosures, each with a redundant SIL3 fail-safe system.

By providing two independent SIL3 fail-safe systems, the required hardware can fit within existing SEM enclosures. The problem now becomes coordinating two independent systems subsea.

Communications and diagnostics

The coordination of two independent SIL3-rated systems requires secure communications. The dual SEM architecture shows SIL 3 "black channel" communications links. (While Ethernet is shown, the communications media itself performs no safety functions and can be considered a "black box," hence the name black channel.) TÜV requirements for black channel-certified communications are designed to detect errors from loss, insertion, repetition, incorrect sequence, corrupted data, delay, and other communications faults.

The SIL 3 black channel allows I/O and diagnostic information to be exchanged between SEM A and SEM B. Proper application design can use this to improve system availability. Each SEM exchanges all system health information, all I/O information, and the status of each field device. If SEM A has a failed field device, it can use the data from a healthy device connected to SEM B to continue operation. This exchange of information over black channel communications improves fault tolerance without impacting operation or safety. If an entire I/O module fails on SEM A, it can still continue operating using data from SEM B.

Achieving fault tolerance

While individual modules within an SEM cannot be replaced, an entire SEM can be removed while the subsea production facility remains in operation with no reduction in SIL rating. Fault-tolerant SIL3 hot swappable subsea control systems are feasible with the proposed architecture.

Hot swapping control modules subsea is only possible if each SEM is in a physically separate location. An ROV obviously must access a failed SEM to replace it. A subsea hot swappable system must include the ability to gracefully remove and reinsert a SEM. This removal and reinsertion must be bumpless, with automatic education and synchronization of the new SEM with current operating data. Not a trivial task, but achievable.

Even though the SEMs are physically separated, they still function as a single system. This requires that the application software in each SEM not only exchange diagnostic information, but be capable of both independent and combined operation.

Redundant topside communications channels may or may not be black channel, but diagnostic data from each SEM can be transmitted topside. This provides redundancy in topside communications and offers an alternative data path if topside links to one SEM fail.

If the SIL 3 black channel Ethernet between each SEM fails, then the topside monitoring system would need to look at the status of each SEM and decide which should continue operation. Loss of communications to a single SEM causes no degradation of system performance since communications topside is maintained through the second SEM. Multiple data pathways increase overall system availability.

If both SEMs lose topside communications, all subsea operations could be immediately shut down, or a predetermined interval could elapse before shutdown occurred.