2012 swapout: equipment removal (2 of 5)

[As always, click photos to see their larger versions.]

Tuesday morning, we got out on the water shortly after 10am, and Anibal Santiago Mercado ("Nibo") maneuvered the boat so that it was anchored into the current with the stern hanging back exactly level with the pylon.  Throughout Tuesday's and Thursday's work it was extremely easy to step from boat to pylon and back again, even despite some moderate wind and waves.  We started with a quick pylon climb to shut off the station's power so that the underwater instruments and cables could be removed without worrying about whether the cable plugs needed to be kept out of the water.

As mentioned previously here, two of the station's four underwater instruments had already failed and been removed, so there remained only two more instruments to remove today:  the shallow light sensor and the deep CTD.  As soon as we saw the deep CTD (see right), it was obvious why the station was without power -- the CTD's plug was snapped off and hanging at a 45-degree angle, with its internal wiring clearly visible.  It's impossible to say for sure whether this happened back in April when the station last lost power, or whether something else failed in April and this CTD connector broke more recently, but it seems reasonable to assume that this broken plug was the cause of the April power failure.  Note too that this CTD did not come back online at the point in time that the station recovered from its February 27th - March 31st downtime, so its communication wiring may have been impacted as early as late February.

The shallow light sensor (see photo at right) was completely covered with growth, including its mounting platform, its connector plug, and its sensing surface.  The last monthly cleaning recorded in this blog was in October of 2011, but there may have been more recent cleanings than this.  This particular location tends to foul more quickly than any other CREWS station and, given that the station was believed to be completely offline beginning in mid-April (and that its instruments would be completely replaced during this swapout operation that would bring it back online) there was no urgent need for instrument cleaning.

However, the station's support lines and chains are now quite considerably fouled (see photo at right), and should receive a thorough cleaning as soon as possible.

The morning's dive took about half an hour and by noon we were done.

Next, I climbed the pylon to disconnect and lower the "brain" (control unit), the three aerial masts with their four instruments (Vaisala integrated weather transmitter or "WXT," anemometer, electronic compass and surface light sensor), and the two battery packs.  As mentioned in yesterday's blog entry, these batteries had been continuously deployed since January of 2006 and in this trip we would be replacing them with new batteries of the same model.  The photo at left shows me lowering one of the two battery packs down to the boat.

One unpleasant surprise was that the Vaisala WXT's custom-made bird deterrent had not survived its maltreatment by the boobies that like to sit on every available surface of this station.  More so than at any other CREWS station, this station's surface instruments are returned to us with a very thick layer of bird guano covering everything.  Since we don't ship new WXT bird deterrents for each swapout (although henceforth, we will do so), and since only one of the deterrent's original six spikes remained (and even this spike was bent flat), this means that the new WXT will have to be deployed with no bird deterrent and its wind/rain measurements may therefore degrade more quickly than usual.

Another problem is that the station's grounding cables, used to electrically connect all the aluminum masts with the electrical ground of the electronics, the lightning diffuser brush, and the grounding plate mounted near the ocean floor, are now falling apart.  The spare grounding cables brought along on this trip were built with a different station design in mind and were too short to be useful for this station, but I repaired/replaced them as far as was possible.

Last on today's agenda was replacing the underwater cables.  This station is unique among CREWS stations in that its internal conduits (which run from just below the solar panels down to the ocean) have their openings beneath the ocean's surface.  This experimental design was not repeated elsewhere because those internal conduits quickly fill with barnacles and other sharp growth and this greatly impedes any effort to add, remove or replace the underwater cables on this station.

A survey of the underwater cables during our dive revealed that there were two cables emerging from the south conduit (which were used for the two deep instruments) and four cables in the north conduit.  This was one more cable than I recalled, but the fourth cable was originally installed for the SAMI pCO2 sensor which has not been present for many years, and the SAMI cable is a simpler (less tough) design compared to the "fishbite" cables used for all other instruments.  Since the SAMI cable was no longer being used and its polyurethane coating was cracking open where it had been exposed to the sunlight, we simply removed and discarded it during this visit.

The five new cables had been prepared by taping their ends (for when they would briefly be submerged during installation) and attaching a distinct number of cable ties (one through five) to each end so that the cable ends could be matched up and the correct cable could be wired to the correct instrument.  During this operation we ran cables #1, #4 and #5 up the northern conduit for the shallow instrument and cables #2 and #3 up the southern conduit for the deep instruments.

We started with the southern conduit.  We taped the end of a "fishtape" spool of hard plastic wire to the clipped ends of the deep instrument cables, and pulled them down the conduits from below.  Throughout this part of the work, I (Mike) stayed atop the pylon and Pamela frequently entered the water on snorkel to pull or push the cables past some obstruction or other.  Nibo stayed in the boat and taped the end of the new cables to the fishtape.  We were pleased and gratified when the two old cables pulled free of the station with almost zero resistance but met with difficulty when trying to pull the two new cables back up the conduit to replace them.  This was mostly due to debris and growth inside the underwater conduit.  After three tries, we eventually managed to feed a single cable up the conduit and then tie the second cable end to the middle of the first cable and feed both cables that way.

Turning to the northern conduit, there was one additional complication -- this conduit also held a green grounding wire (see photo at right, taken during Thursday's work) that ran from the mast/electronics/brush grounding system to the grounding plate at the base of the pylon.  Even if we'd wanted to try feeding all three new cables up to the top of the pylon (which we did not, given the frustration we'd had with trying to feed two cables at once up the other side), we did not want to pull out all four cables from the conduit at once for fear that they might be wrapped around the grounding wire and might cause it to pull loose or break.

So we decided to feed the new cables one by one.  We started by pulling out the SAMI cable, since it was distinguishable from the other three at both the top and the bottom, and fed a new cable to replace it.  Then it occurred to us that we could pull free a second cable at random without using it to feed a new cable, thereby freeing up some space in the conduit.  After that we replaced the last two cables one by one.  This took much longer than expected and we didn't leave the station that afternoon until almost 5:30pm.  But we were well-poised for the rest of the week's work.

[All photos taken by Pamela Fletcher.]

2012 swapout: preparations and travel (1 of 5)

On Monday, August 27th, 2012, a team from NOAA's Atlantic Oceanographic and Meteorological Laboratory (AOML) traveled to La Parguera, Lajas, Puerto Rico, in order to spend the week swapping out equipment on the Coral Reef Early Warning System (CREWS) station at Media Luna Reef.  The team consisted of Pamela Fletcher (University of Florida / SeaGrant / AOML) and Mike Jankulak (University of Miami / AOML).

Normally a CREWS station is visited once every year and its oceanographic and meteorological instruments are swapped out, its programming is updated, and its locally-stored data are recovered.  This trip was different from our usual swapouts in several important ways:

• The last swapout operation at this station was in November of 2010, nearly two years ago.  This current trip had been in the planning stages since January of 2012 but personnel shortages had made it impossible before now.  Therefore the station's instruments, cables and other hardware had had almost twice as long as usual to degrade and might be expected to be in worse condition than usual.
• Since the last swapout, this station had suffered three power failures.  The first failure (described here) was on July 14th, 2011, and was remotely diagnosed as a failure of the deep light sensor.  Fortunately the University of Puerto Rico was able to react quickly and the failed sensor was removed on July 15th, and station operations started coming back online two days later.
• The second failure was on February 27th, 2012, and this time the shallow CTD was indicated by the data record to be the most likely cause.  The UPR folks were too busy to visit the station until another team from AOML arrived in the week of March 12th - 16th to replace the station's support lines.  The shallow CTD was removed during these operations and the station regained power a bit more than two weeks later, on March 31st.  At the time that operations resumed, the deep CTD was found to be offline as well.
• Then after another two weeks, the station lost power a third time and never recovered.  The cause of this third power failure remained unknown up until the time of this current August/2012 swapout operation.  Since there had been this many problems with power failures, and since this station's electronics and cables were originally deployed more than six years ago, we made the decision to replace all of the station's underwater cables on this trip.
• In September of 2011, after the first of the three power failures described above, I conducted an analysis of the station's recent record of dropped satellite transmissions.  This analysis revealed that the station's "daylight" transmission success rate was about 86%, which was lower than we'd like but still not that bad, but its "dark" transmission success rate was only 57%, which was terrible.  This strongly suggested that the station was more likely to supply enough power to the transmitter while its solar panels were providing additional power to the system, whereas when powered by only the batteries there was often not sufficient power.  Given that the batteries were then halfway through their sixth year of deployment (and an additional year has now passed since that analysis took place), and given additionally that the batteries had been fully drained by more than one short-circuit event, we decided to replace the batteries on this current swapout trip.
• One final difference in this year's swapout was that we had elected to try a different make of CTD in this year's deployment.  Instead of deploying our usual RDI/Teledyne (formerly Falmouth) CTDs, this year we elected to deployed two refurbished SeaBird Microcat SMP37 CTDs.  These CTDs require different programming and their maintenance needs are different from the Teledyne CTDs.
• On a minor note, this swapout would also be the first such operation to be conducted with the new CREWS travel laptop, following the failure of the old laptop's battery supply.  The new laptop runs Windows 7 compared to the old laptop's Windows XP, and the old laptop had a built-in serial port whereas the new laptop requires the use of a USB-to-Serial adaptor.

Arrangements for this week of work began in July, when our replacement instruments and tools were shipped from Miami to UPR.  Air travel and hotel arrangements were finalized in early August.  Coordinating with UPR's Francisco Pagan, we arranged for boat and diving support, with boat operations to be carried out by Anibal Santiago Mercado, for the three days from Tuesday to Thursday.

There was some last-minute doubt about the timing of this trip with the formation of TS Isaac and its effects on UPRM (in the week before this operation) and Miami (in the days before we were scheduled to fly out of Fort Lauderdale).  In the end, however, the trip was able to go ahead as planned and we (Pamela and Mike) arrived in La Parguera Monday evening, ready for work to begin on Tuesday morning.

Complete Cleaning in October

Today (Monday, October 3rd, 2011), a complete cleaning (CTD validation, cleaned all sensors, chains, lines, and spar) was done at the CREWS Station in La Parguera, Puerto Rico, by Wessley Merten. The groundtruth validation sensor was placed on at 0945 and was removed at 1324.

Basic Cleaning in August

Yesterday, Tuesday August 30th, Wess Merten conducted a thorough basic cleaning and assessment of the CREWs station in La Parguera. I conducted a surface inspection, underwater inspection/cleaning, optical sensor cleaning, and a CTD sensor cleaning.  I also very carefully removed some biofouling that has accumulated on some of the cables.  In addition, all four brass filters were replaced on the CTD's.  After more than 3 hours underwater the station looks great!  A complete cleaning will be conducted next week. 

Light Sensor Failure, Station Downtime

The CREWS station at Media Luna reef (near La Parguera, Puerto Rico) experienced a complete power loss on July 14th and has since been recovering slowly. This is strikingly similar to an incident that took place in April of 2010, just over one year ago.

In both cases it is believed that the integrity of the deeper of the two underwater light sensors was compromised and it became flooded. This led initially to the loss of data messages from the sensor but, as the flooding became more severe, resulted in the short-circuiting of the station's entire power supply and therefore a complete loss of power. Following a remote analysis which in both cases identified the deep light sensor at the likely cause, the instrument was retrieved to the surface and removed, and its cable end securely plugged. At this point the station was operable once again but completely drained of battery reserves, and it took (is taking) several weeks for it to replenish its battery reserves by means of its solar panels. A detailed timeline of the two incidents follows:

In 2010, the instrument failed on April 14th and the station lost power just over a day later on the 15th. Unfortunately we here in Miami didn't notice the station's failure before the weekend and so we let three days go by before first contacting UPR on April 19th, a Monday. Wess made several trips to the station and took photos but he couldn't remove the failed sensor until we shipped out a dummy plug for the cable. This meant that it was over a week later that the instrument was removed (April 23rd) and another 4 days before the station started transmitting again (April 27th). The total downtime was almost 12 days. Following the resumption of station transmissions, it took about 17 more days before the station's voltages returned to normal levels.

Compare this to what happened this year: the deep light sensor failed on July 12th (Tuesday) and I noticed this by chance less than two days later on July 14th (Thursday). This means we were able to request help from UPR even before the station lost power, which it did later on the 14th. Francisco responded quickly and by the afternoon of the 15th (Friday) he reported that the instrument had been removed from the station. With so little time between station failure and repair, the station started transmitting again after only two days, for a total downtime of less than 3 days. As of this writing (12 days since transmissions resumed), station voltages have not yet returned to normal levels and it may be another week before the station can maintain steady and reliable transmissions 24 hours a day.

At right is a graph of the station's battery levels so far in 2011, with voltage plotted against day-of-year. [Please click on the image for a larger version.] Note the usual diurnal cycle of voltages, rising to almost 14 V every day in the sunlight and falling to just below 12.75 V overnight. There was an unexplained disruption of this pattern in the month leading up to the sensor's failure, which may or may not have been caused by the problems with this sensor. The graph clearly shows that the station battery levels have not yet returned to normal.

Until we can inspect the recovered light sensor, it is too soon to say whether there may be some specific and ongoing risk to this type of sensor at this particular depth at this particular station. The only other similar incident in CREWS history was judged to be the result of a failure at the connection point of the instrument cable, possibly compounded by a cable insufficiently tied down and therefore subject to a violent whipping back and forth in the ocean currents (this was at the Jamaica CREWS station, in 2008). However, the 2010 sensor failure at La Parguera was of a different nature: in that case, the failure point was a puncture of the sensing surface of the instrument, on top. If this month's failure was also due to a puncture at the top of the sensor, it may signal that our deployment strategy for these light sensors should be reviewed.

The AOML team here in Miami are extremely grateful to our UPR colleagues for their super-quick response to this incident. A reduction of downtime from 12 days in 2010 to less than 3 days this year is an amazing thing to report.

LPPR1 Seabird Electronic tempearture loggers

Recently, the Coral Reef Early Warning System or CREWs station in La Parguera had two additional instruments on-board to collect data for researches at Rosenstiel School of Marine and Atmospheric Science.  Known as Seabird Electronic tempeature loggers (currently in beta field trails), two of these instruments were placed just below the surface and ~18" down to monitor diurnal warming of Parguera's inshore shallow water.  The following summation of their research was written by Xiaofang Zhu a graduate assistant at the Rosenstiel School of Marine and Atmospheric Science:

Coral reefs are very sensitive to thermal stresses and recent studies have suggested the daily warming phenomenon in the shallow water, along with weekly temperature averages can affect coral bleaching. This study is interested in the detailed physical processes of diurnal warming, including the timing, amplitude and vertical temperature profiles. To reach this goal, a Seabird temperature logger is added along with the existing 2 CTDs to help understand the warming phenomenon at 3 different depths, and to construct a vertical temperature profile.

Below is a graph (click for larger view) of the agreement between the existing CTD temperature sensors and the Seabird Electronics temperature loggers as plotted by Xiaofang.

Unfortunately, both Seabird instruments are not active on the station at this moment due to technical difficulties.  I will keep you posted when they are back on-board the station.

Complete Cleaning in May

Today (Tuesday, May3rd, 2011), a complete cleaning (CTD validation, cleaned all sensors, chains, lines, and spar) was done at the CREWS Station in La Parguera, Puerto Rico, by Wessley Merten. The groundtruth validation sensor was placed on at 0900 and was removed at 1230.  All surface and subsurface sensors were standing proud to the sea.  All rigging, lines, baseplates, sensors, and the spar were cleaned.  The waters were really clear today which made a clean station look really great.