2012 swapout: reinstallation (4 of 5)

[As always, click photos to see their larger versions.]

On a happy note, I woke up early this morning with my head spinning with ideas about the two CTDs that would not communicate, and the idea came to me that perhaps their "instrument transmit" and "instrument receive" wires (Tx and Rx) had been reversed in our specially-made pigtail adapters.  If that was the case then I'd surely had to use "null modems" to connect to them in my office in January but perhaps I'd done just that and hadn't remembered it, due to the long delay between testing and deployment.  So when we arrived at the lab, the first thing I tried was to connect to the CTDs via null modems and hurrah!, that was successful.  Fortunately I had two null modems with me that I could wire into the brain connections at the top of the station later that day.

With that short delay we were still ready to leave the dock by 9am.  Upon arriving at the station I connected the five underwater instruments, on the boat, to the five new cables with cable-tie markings as follows:

1. "Groundtruth" CT, north side
2. Deep Light Sensor, south side
3. Deep CTD, south side
4. Shallow Light Sensor, north side
5. Shallow CTD, north side 

By 10am we were ready for our first dive, to install the instruments on the station.  The "groundtruth" CT would be temporarily cable-tied in place (see photo at right, taken before cables were tied down).  Once it was proven to be communicating correctly, it would be disconnected and its dummy-plugged cable end would be cable-tied loosely just below where the chains attached to the pylon near the ocean surface.

The CTDs were screwed down to two mounting brackets each, and each mounting bracket would eventually be hose-clamped to the pylon in two places.  During initial installation we used only two hose clamps per CTD instead of four (see photo of the deep CTD at left) in case we later had to retrieve the CTDs for additional work on the boat.

During this dive we also removed the last two of the old cables, for these had been tangled or cable-tied and we'd been unable to recover them on Tuesday on snorkel alone.  We also did a brief swim around the periphery of the station, going clockwise from pin to pin around the station.  In all this dive took 49 minutes.

From about 11am to 3pm I climbed the station and installed everything that had been removed on Tuesday, but in reverse:  the new battery packs, the three aerial masts, and the "brain."  Above is a picture of the inside of the "brain" chamber at the top of the pylon before reinstalling the batteries; right is a picture of me installing the Vaisala WXT.  Before installing the brain I first had to wire six plugs (four wires each) to the ends of the five new underwater cables and the new cable I was using for the surface light sensor.  After the brain was installed and everything was connected, I flipped the power switch and turned it on.

Next I returned to the boat to connect from my laptop (via radio) to the brain to ensure that all instruments were working properly.  However, I could get no connection at all and I had not remembered noticing any lights on the brain when I turned it on, so I decided to climb again and check if any obvious connections were unplugged.  I did so, powering off the brain, checking connections, and then carefully watching the lights when powering it back on.  I could see the serial ports briefly light up red on startup, and the radio unit flashing red to indicate that it was waiting for a connection, so I returned to the boat and tried again.  When there was still no connection, I switched radios on the boat and tried a different unit, and this time I was able to connect.  At this point all instruments were reporting correctly except, alas, the two SeaBird CTDs.  However I knew that the datalogger programming was written to set up the CTDs to report every six minutes, and at several points it had to wait until the exact six-minute mark before proceeding, so it might take six, twelve or even eighteen minutes before the CTDs would start reporting.  I returned to the pylon to plug up its external openings and tie down all loose cables, in preparation for closing it up, while we waited to see if the CTDs would wake up on their own.  When I next checked the CTDs they were fine, so I finished closing up the station top and removed the safety line and rungs.

One further note about the station top:  the bottom of the transmitter's satellite antenna (see photo at right) is showing signs of peeling paint and might be suffering moisture damage.  We should keep a careful eye on the transmission success rate and perhaps replace this antenna on our next trip regardless of whether it appears to be working normally or not.

At 4:30pm Pamela and I began our final dive to retrieve the groundtruth CT and lock down the other four instruments, and their cables, more securely.  The deep light sensor (whose platform is of a different design than that used for the shallow light sensor) was sitting loosely in its mount so we attempted to gum up the loose space with aquaseal.  A note for future deployments: normally I don't use mastic tape at the point where the light sensors are encircled by their mounts but for this station's deep light sensor we should make an exception.

Pamela noted where the station's lone "warning" buoy (originally there were four, now only one remains) has fouled considerably and its chain is covered with barnacles that appear to pose a chafing danger to the spectra line on that support chain (see photo at right).  We would suggest removing or replacing this buoy.

I will also include a photo (at left) of the bottom portion of the station, to show how much bio-fouling there has been since these spectra line were replaced in March.

A final underwater photo (at right) illustrates very well how completely fouled the chains are.  You mostly cannot see where one link ends and another link begins, and in most places you could not stick your finger through the hole in these chains (much less determine whether they still have sacrificial zincs installed).  Our time at the station was severely limited on this trip and we did not have any time for cleaning the chains or spectra, but this type of cleaning should be done with very high priority once it is determined who will be doing the monthly cleanings from here on out.

We took some post-swapout photos of the station (like this one, below) and then returned to the dock by 5:30pm, where we spent the next two hours cleaning our gear and packing our instruments and supplies for their return shipment to Miami.

[Photos taken by Pamela Fletcher and Mike Jankulak.]

2012 swapout: land-based work (3 of 5)

The agenda for Wednesday was to prepare the underwater instruments for deployment and then load up the boat and return to the station.  After installing the underwater instruments, we could then conduct an informal "reef check" exercise to be led by Pamela in the vicinity of the station.  We had decided not to follow strict reef check protocols because of the unsuitable topology of the ocean floor in that area (on this trip we were both limited to a maximum depth of 30 feet), but rather to "rehearse" reef check methods for future use and at the same time to collect some notes about the biological environment of the station.  There was no plan to do any pylon climbing on this day, instead we would return to the lab after diving so that I could reprogram/rewire the instruments so that we would be ready for Thursday's reinstallation and activation of the station.

Prepping the instruments mostly involved installing the batteries in the SeaBird CTDs and connecting to them to set their dates and times and verify that they were working.  We had planned to leave the dock by 9:30am but unfortunately the CTDs would not respond to any communication efforts, either one of them, and by 10am I decided to cancel the day's boat work and diving to devote myself to the electronics entirely.

By noon I had consulted with SeaBird technical support and what AOML personnel could be reached, and I was still unsuccessful in communicating with the CTDs.  [Note:  tech support's first instruction to me was to connect to the SeaBird CTDs with their own SeaBird communications cable, which to my knowledge we do not have.  If we continue to deploy these SeaBird CTDs we should definitely acquire and travel with one of these cables for use in debugging problems like these.]

I then turned to the other electronics work:  downloading the station's local memory files, uploading the new datalogger program, reprogramming the serial port units (SIO4s), replacing the four aerial instruments on their three masts (the anemometer and electronic compass share the same mast), replacing old cables, and testing all plugs to ensure that their wire/pin connections were unbroken.  I also cut away the duct tape from the old battery packs and taped up the new batteries with spectra-line handles in the same way.

During this work, one of the "brain" fuses was discovered to have blown, and was replaced.  It's not clear when this happened.

While working away in the lab I managed to meet up with Wess Merten, who has been doing the monthly station cleanings for several years, and asked him for three things:  one, the "groundtruth" CT sensor which I needed to connect on Thursday to ensure that its cable and wiring were properly connected; two, the deep light sensor that had been removed in July of 2011; and three, the shallow CTD that had been removed in March of 2012.  He was able to locate the first two sensors immediately but he did not know the exact whereabouts of the CTD and said that he would look for it.

By the end of the day everything was ready for deployment but the CTDs were still non-operational.  I wondered if my new laptop's USB-to-Serial adapter was somehow failing to communicate with the instruments, although it worked fine with the light sensors and the datalogger.  [Also, I connected to the shallow light sensor that we'd retrieved the day before and it still appeared to be operational.]  There was no good explanation for why (1) both CTDs were unable to communicate when (2) they had both been operating normally during testing in my office last January.  I decided we could try deploying them anyhow, reasoning that if the USB-to-Serial adapter was somehow to blame then perhaps the direct connection to our datalogger would work as it had in January.  In the worst case, if neither CTD would work, we could permanently install the "groundtruth" CT until such time as replacement CTDs could be installed.

2012 swapout: equipment removal (2 of 5)

[As always, click photos to see their larger versions.]

Tuesday morning, we got out on the water shortly after 10am, and Anibal Santiago Mercado ("Nibo") maneuvered the boat so that it was anchored into the current with the stern hanging back exactly level with the pylon.  Throughout Tuesday's and Thursday's work it was extremely easy to step from boat to pylon and back again, even despite some moderate wind and waves.  We started with a quick pylon climb to shut off the station's power so that the underwater instruments and cables could be removed without worrying about whether the cable plugs needed to be kept out of the water.

As mentioned previously here, two of the station's four underwater instruments had already failed and been removed, so there remained only two more instruments to remove today:  the shallow light sensor and the deep CTD.  As soon as we saw the deep CTD (see right), it was obvious why the station was without power -- the CTD's plug was snapped off and hanging at a 45-degree angle, with its internal wiring clearly visible.  It's impossible to say for sure whether this happened back in April when the station last lost power, or whether something else failed in April and this CTD connector broke more recently, but it seems reasonable to assume that this broken plug was the cause of the April power failure.  Note too that this CTD did not come back online at the point in time that the station recovered from its February 27th - March 31st downtime, so its communication wiring may have been impacted as early as late February.

The shallow light sensor (see photo at right) was completely covered with growth, including its mounting platform, its connector plug, and its sensing surface.  The last monthly cleaning recorded in this blog was in October of 2011, but there may have been more recent cleanings than this.  This particular location tends to foul more quickly than any other CREWS station and, given that the station was believed to be completely offline beginning in mid-April (and that its instruments would be completely replaced during this swapout operation that would bring it back online) there was no urgent need for instrument cleaning.

However, the station's support lines and chains are now quite considerably fouled (see photo at right), and should receive a thorough cleaning as soon as possible.

The morning's dive took about half an hour and by noon we were done.

Next, I climbed the pylon to disconnect and lower the "brain" (control unit), the three aerial masts with their four instruments (Vaisala integrated weather transmitter or "WXT," anemometer, electronic compass and surface light sensor), and the two battery packs.  As mentioned in yesterday's blog entry, these batteries had been continuously deployed since January of 2006 and in this trip we would be replacing them with new batteries of the same model.  The photo at left shows me lowering one of the two battery packs down to the boat.

One unpleasant surprise was that the Vaisala WXT's custom-made bird deterrent had not survived its maltreatment by the boobies that like to sit on every available surface of this station.  More so than at any other CREWS station, this station's surface instruments are returned to us with a very thick layer of bird guano covering everything.  Since we don't ship new WXT bird deterrents for each swapout (although henceforth, we will do so), and since only one of the deterrent's original six spikes remained (and even this spike was bent flat), this means that the new WXT will have to be deployed with no bird deterrent and its wind/rain measurements may therefore degrade more quickly than usual.

Another problem is that the station's grounding cables, used to electrically connect all the aluminum masts with the electrical ground of the electronics, the lightning diffuser brush, and the grounding plate mounted near the ocean floor, are now falling apart.  The spare grounding cables brought along on this trip were built with a different station design in mind and were too short to be useful for this station, but I repaired/replaced them as far as was possible.

Last on today's agenda was replacing the underwater cables.  This station is unique among CREWS stations in that its internal conduits (which run from just below the solar panels down to the ocean) have their openings beneath the ocean's surface.  This experimental design was not repeated elsewhere because those internal conduits quickly fill with barnacles and other sharp growth and this greatly impedes any effort to add, remove or replace the underwater cables on this station.

A survey of the underwater cables during our dive revealed that there were two cables emerging from the south conduit (which were used for the two deep instruments) and four cables in the north conduit.  This was one more cable than I recalled, but the fourth cable was originally installed for the SAMI pCO2 sensor which has not been present for many years, and the SAMI cable is a simpler (less tough) design compared to the "fishbite" cables used for all other instruments.  Since the SAMI cable was no longer being used and its polyurethane coating was cracking open where it had been exposed to the sunlight, we simply removed and discarded it during this visit.

The five new cables had been prepared by taping their ends (for when they would briefly be submerged during installation) and attaching a distinct number of cable ties (one through five) to each end so that the cable ends could be matched up and the correct cable could be wired to the correct instrument.  During this operation we ran cables #1, #4 and #5 up the northern conduit for the shallow instrument and cables #2 and #3 up the southern conduit for the deep instruments.

We started with the southern conduit.  We taped the end of a "fishtape" spool of hard plastic wire to the clipped ends of the deep instrument cables, and pulled them down the conduits from below.  Throughout this part of the work, I (Mike) stayed atop the pylon and Pamela frequently entered the water on snorkel to pull or push the cables past some obstruction or other.  Nibo stayed in the boat and taped the end of the new cables to the fishtape.  We were pleased and gratified when the two old cables pulled free of the station with almost zero resistance but met with difficulty when trying to pull the two new cables back up the conduit to replace them.  This was mostly due to debris and growth inside the underwater conduit.  After three tries, we eventually managed to feed a single cable up the conduit and then tie the second cable end to the middle of the first cable and feed both cables that way.

Turning to the northern conduit, there was one additional complication -- this conduit also held a green grounding wire (see photo at right, taken during Thursday's work) that ran from the mast/electronics/brush grounding system to the grounding plate at the base of the pylon.  Even if we'd wanted to try feeding all three new cables up to the top of the pylon (which we did not, given the frustration we'd had with trying to feed two cables at once up the other side), we did not want to pull out all four cables from the conduit at once for fear that they might be wrapped around the grounding wire and might cause it to pull loose or break.

So we decided to feed the new cables one by one.  We started by pulling out the SAMI cable, since it was distinguishable from the other three at both the top and the bottom, and fed a new cable to replace it.  Then it occurred to us that we could pull free a second cable at random without using it to feed a new cable, thereby freeing up some space in the conduit.  After that we replaced the last two cables one by one.  This took much longer than expected and we didn't leave the station that afternoon until almost 5:30pm.  But we were well-poised for the rest of the week's work.

[All photos taken by Pamela Fletcher.]

2012 swapout: preparations and travel (1 of 5)

On Monday, August 27th, 2012, a team from NOAA's Atlantic Oceanographic and Meteorological Laboratory (AOML) traveled to La Parguera, Lajas, Puerto Rico, in order to spend the week swapping out equipment on the Coral Reef Early Warning System (CREWS) station at Media Luna Reef.  The team consisted of Pamela Fletcher (University of Florida / SeaGrant / AOML) and Mike Jankulak (University of Miami / AOML).

Normally a CREWS station is visited once every year and its oceanographic and meteorological instruments are swapped out, its programming is updated, and its locally-stored data are recovered.  This trip was different from our usual swapouts in several important ways:

• The last swapout operation at this station was in November of 2010, nearly two years ago.  This current trip had been in the planning stages since January of 2012 but personnel shortages had made it impossible before now.  Therefore the station's instruments, cables and other hardware had had almost twice as long as usual to degrade and might be expected to be in worse condition than usual.
• Since the last swapout, this station had suffered three power failures.  The first failure (described here) was on July 14th, 2011, and was remotely diagnosed as a failure of the deep light sensor.  Fortunately the University of Puerto Rico was able to react quickly and the failed sensor was removed on July 15th, and station operations started coming back online two days later.
• The second failure was on February 27th, 2012, and this time the shallow CTD was indicated by the data record to be the most likely cause.  The UPR folks were too busy to visit the station until another team from AOML arrived in the week of March 12th - 16th to replace the station's support lines.  The shallow CTD was removed during these operations and the station regained power a bit more than two weeks later, on March 31st.  At the time that operations resumed, the deep CTD was found to be offline as well.
• Then after another two weeks, the station lost power a third time and never recovered.  The cause of this third power failure remained unknown up until the time of this current August/2012 swapout operation.  Since there had been this many problems with power failures, and since this station's electronics and cables were originally deployed more than six years ago, we made the decision to replace all of the station's underwater cables on this trip.
• In September of 2011, after the first of the three power failures described above, I conducted an analysis of the station's recent record of dropped satellite transmissions.  This analysis revealed that the station's "daylight" transmission success rate was about 86%, which was lower than we'd like but still not that bad, but its "dark" transmission success rate was only 57%, which was terrible.  This strongly suggested that the station was more likely to supply enough power to the transmitter while its solar panels were providing additional power to the system, whereas when powered by only the batteries there was often not sufficient power.  Given that the batteries were then halfway through their sixth year of deployment (and an additional year has now passed since that analysis took place), and given additionally that the batteries had been fully drained by more than one short-circuit event, we decided to replace the batteries on this current swapout trip.
• One final difference in this year's swapout was that we had elected to try a different make of CTD in this year's deployment.  Instead of deploying our usual RDI/Teledyne (formerly Falmouth) CTDs, this year we elected to deployed two refurbished SeaBird Microcat SMP37 CTDs.  These CTDs require different programming and their maintenance needs are different from the Teledyne CTDs.
• On a minor note, this swapout would also be the first such operation to be conducted with the new CREWS travel laptop, following the failure of the old laptop's battery supply.  The new laptop runs Windows 7 compared to the old laptop's Windows XP, and the old laptop had a built-in serial port whereas the new laptop requires the use of a USB-to-Serial adaptor.

Arrangements for this week of work began in July, when our replacement instruments and tools were shipped from Miami to UPR.  Air travel and hotel arrangements were finalized in early August.  Coordinating with UPR's Francisco Pagan, we arranged for boat and diving support, with boat operations to be carried out by Anibal Santiago Mercado, for the three days from Tuesday to Thursday.

There was some last-minute doubt about the timing of this trip with the formation of TS Isaac and its effects on UPRM (in the week before this operation) and Miami (in the days before we were scheduled to fly out of Fort Lauderdale).  In the end, however, the trip was able to go ahead as planned and we (Pamela and Mike) arrived in La Parguera Monday evening, ready for work to begin on Tuesday morning.