Friday, July 29, 2011

Light Sensor Failure, Station Downtime

The CREWS station at Media Luna reef (near La Parguera, Puerto Rico) experienced a complete power loss on July 14th and has since been recovering slowly. This is strikingly similar to an incident that took place in April of 2010, just over one year ago.

In both cases it is believed that the integrity of the deeper of the two underwater light sensors was compromised and it became flooded. This led initially to the loss of data messages from the sensor but, as the flooding became more severe, resulted in the short-circuiting of the station's entire power supply and therefore a complete loss of power. Following a remote analysis which in both cases identified the deep light sensor at the likely cause, the instrument was retrieved to the surface and removed, and its cable end securely plugged. At this point the station was operable once again but completely drained of battery reserves, and it took (is taking) several weeks for it to replenish its battery reserves by means of its solar panels. A detailed timeline of the two incidents follows:

In 2010, the instrument failed on April 14th and the station lost power just over a day later on the 15th. Unfortunately we here in Miami didn't notice the station's failure before the weekend and so we let three days go by before first contacting UPR on April 19th, a Monday. Wess made several trips to the station and took photos but he couldn't remove the failed sensor until we shipped out a dummy plug for the cable. This meant that it was over a week later that the instrument was removed (April 23rd) and another 4 days before the station started transmitting again (April 27th). The total downtime was almost 12 days. Following the resumption of station transmissions, it took about 17 more days before the station's voltages returned to normal levels.

Compare this to what happened this year: the deep light sensor failed on July 12th (Tuesday) and I noticed this by chance less than two days later on July 14th (Thursday). This means we were able to request help from UPR even before the station lost power, which it did later on the 14th. Francisco responded quickly and by the afternoon of the 15th (Friday) he reported that the instrument had been removed from the station. With so little time between station failure and repair, the station started transmitting again after only two days, for a total downtime of less than 3 days. As of this writing (12 days since transmissions resumed), station voltages have not yet returned to normal levels and it may be another week before the station can maintain steady and reliable transmissions 24 hours a day.

At right is a graph of the station's battery levels so far in 2011, with voltage plotted against day-of-year. [Please click on the image for a larger version.] Note the usual diurnal cycle of voltages, rising to almost 14 V every day in the sunlight and falling to just below 12.75 V overnight. There was an unexplained disruption of this pattern in the month leading up to the sensor's failure, which may or may not have been caused by the problems with this sensor. The graph clearly shows that the station battery levels have not yet returned to normal.

Until we can inspect the recovered light sensor, it is too soon to say whether there may be some specific and ongoing risk to this type of sensor at this particular depth at this particular station. The only other similar incident in CREWS history was judged to be the result of a failure at the connection point of the instrument cable, possibly compounded by a cable insufficiently tied down and therefore subject to a violent whipping back and forth in the ocean currents (this was at the Jamaica CREWS station, in 2008). However, the 2010 sensor failure at La Parguera was of a different nature: in that case, the failure point was a puncture of the sensing surface of the instrument, on top. If this month's failure was also due to a puncture at the top of the sensor, it may signal that our deployment strategy for these light sensors should be reviewed.

The AOML team here in Miami are extremely grateful to our UPR colleagues for their super-quick response to this incident. A reduction of downtime from 12 days in 2010 to less than 3 days this year is an amazing thing to report.