Station is offline until further notice

The LPPR1 station as seen in 2013.

The CREWS station located near Media Luna Reef in La Parguera, PR, will be offline until further notice.  The station has experienced numerous communications and power problems over the last few years and diagnoses of these problems remain elusive.  The last repair attempt, described in this blog post, resulted in the known failure of both CTDs and a suboptimal placement of the deep light sensor after a key part of its mounting platform broke.  This was shortly followed by yet another loss of communications and a probable loss of power, on August 31st, 2013.

This CREWS station was physically deployed on October 18th, 2005 and first went live on January 15th, 2006 following the installation of its electronics, sensors and solar panels.  This station has operated for nearly eight years, which far exceeds the station's expected lifetime, estimated by some to have been approximately five years.

Given that repeated repair attempts have failed to correct the mysterious disconnects between the station electronics, solar panels and rechargeable batteries, the station must now be considered irreparable, short of recovering the station to land for extensive analysis and repair.  Rather than bring the station to land and then redeploy it in its current form, plans are being made to replace this station with a CREWS buoy similar to those that have recently been deployed in Belize, Tobago (BUTO1 and ARTO1), Little Cayman (CCMI2) and Barbados (DRBB1).  Note that this buoy design is also a distant cousin of the MApCO2 buoy currently operating nearby at Cayo Enrique.  The first stage of this work, that of station removal, could possibly begin as early as late 2014 or early 2015.

Further information on this station's removal and replacement will be posted on this blog as it becomes available.

complete cleaning

[From an email from Diana Beltran, November 18, 2013 5:02 PM EST.]

Last Wednesday November 13/13, we made complete cleaning station. We started connecting the CTD at 8:05 am, then start cleaning at 10:10 am (sensors, center pole, and chains). We completed cleaning at 10:55 am. Then at 1:40 pm we picked up the CTD.

The sensors were all covered with filamentous algae. The chains and the central column were partially covered with Bivalves, sponges, filamentous algae, some corals, etc.

Diana
(posted by Mike Jankulak)

CREWS La Parguera

[From an email from Diana Beltran, October 31, 2013 4:32 PM EDT.]

On October 8 and today 31 /2013, I made sensors cleaning station. We started at 8:00 am. The sensors were all covered with filamentous algae and some white calcareous algae. The chains and the central column were all covered with Bivalves, sponges, filamentous algae, some corals, etc. I tried to clean the chains but has been hard work, I cleaned three so far, so I will continue cleaning them.

Diana
(posted by Mike Jankulak)

post-mortem: what went wrong last week in Puerto Rico

During the week of August 26th - 30th, two members of the CHAMP team (Rachel Kotkowski and Mike Jankulak) visited the CREWS station at La Parguera, PR.  Our goals were:

• Identify and fix whatever had taken the station offline last March.
• Swap out all instruments, since it is now one year after last August's swapout.
• Move the "deep" instruments several metres higher.

Unfortunately we encountered problems from start to finish and fell short of nearly all of these goals, to wit:

• The station lost power again on August 31st, only two days after reinstallation.
• Both CTDs on the station failed the same day they were installed.
• The mounting platform for the deep BIC (light sensor) is broken and the sensor is currently hose-clamped directly to the pylon.
• Our UPR CREWS contact was unexpectedly absent and we had to work across a language barrier with her replacement.

Arguably the station is worse off compared to how we found it at the beginning of the week, and we still have no clear idea of what the problems are.  Below I will go into more detail about each of the issues listed above.

First, the power failure.  I had last visited the station in January and in the weeks following that visit I noticed that it seemed to be running off of battery power only, with no daily recharge from the solar panels.  At the time I guessed maybe that the solar panel connector-plug had not been fully plugged in at the top of the pylon, or perhaps one of the connector's pins had broken off.  Last Tuesday I recovered the "brain" (control package) and both sets of batteries from the station and brought them back to land.  The batteries I found to be drained to about 3.5V (compared to a normal range from 12V to 14V), and I connected them to a power supply for a quick charge back to near-normal levels.

However, I could find no evidence of loose pins, broken wires, or failed connectors, and both fuses in the brain were intact.  Far more confusing was the evidence in the logger's memory cache of what had been going on after the transmissions failed in mid-March.  As I'd thought, the station from January to March appeared to be slowly draining its batteries until levels were too low for transmitting.  But after a few weeks of no activity at all, the station in early April appeared (by evidence of its power levels) to have developed the opposite problem from before.  Instead of batteries connected without solar panels, from April to August the station appeared to have solar panels connected without batteries.  The new pattern of voltages began April 4th, which is two days after one of the monthly cleanings on April 2nd.  I do not know if there is any logical connection between these two events.

Click on graph to see a larger version.

Above is a graph of power levels since the beginning of 2013 (the x-axis is labeled with the day of year, from 0 to 365, and the y-axis is voltage).  There is a normal diurnal voltage pattern until my visit at the end of January, after which there are only a few days when battery charging occurs but mostly the power levels steadily decline.  In mid-March the power levels fall below 9V and all station activity ceases.  However, on April 4th the station power levels start fluctuating wildly between 0V at night to more than 17V in the daytime, which is a pattern I have seen when a station has lost its connection to its batteries but still has solar panels plugged in.  In this situation the station generally doesn't have a steady enough power supply to complete even one full transmission, although we did receive just one report on June 2nd (which led to a lot of head-scratching, since at that point we believed the station to be entirely without power).

As I've said, I could not find any "smoking gun" to explain these problems.  The evidence suggests that there has been intermittently faulty connections to the batteries and the solar panels, but not at the same time.  I'd brought a new charger-controller with me and, since this was one element that connects to both batteries and solar panels, I replaced it in the hopes that improvements would follow.  I also speculated that perhaps some of the brain hardware (such as the on/off switch) might be causing the problem.  The "brain" is made up of a backboard of fibreglass with many simple hardware elements screwed or soldered on permanently.  I have often replaced the main electronics components (datalogger, SIO4 serial-port units, satellite transmitter, RF radio) but I have never replaced the baseline hardware elements nor do I have the tools or expertise to do so.  To my knowledge none of these simpler hardware elements have ever failed at a CREWS station before now.

In any case I reinstalled the brain on Thursday, knowing that it would likely take several days of charging to know whether the solar panels and batteries were working properly again.  In fact it took only two days for transmissions to fail this time, probably because the batteries had been drained for months and two days of charging them off my small power supply was not enough to keep them going very long.

I am unsure what might be required to fix the power problems now.  One conservative approach would be to build an entirely new "brain" package, including all switches, connectors, screw-down boards and fuses.  Even this, however, is not guaranteed to help if there is a wiring problem with the permanently-installed solar panels on the station.  Also the wiring of the solar panels at this station is different from all other CREWS stations past or present and replacing the brain entirely might require careful examination and testing of those solar panels (or in the worst case, replacement of the solar panels).

Next, the CTDs:  As usual I reinstalled all station equipment and powered it on, then returned to the boat to connect by radio from my laptop to the station to verify that everything was working properly.  The CTDs have a more complicated program than most instruments -- the datalogger essentially "programs" them to run in interval mode and sets their clock according to its GPS-derived time.  The CTDs when online will spontaneously offer up new readings every 6 minutes but it often takes 6, 12 or even 18 minutes after startup before they start reporting.  So I checked in every 6 minutes and after a few cycles both CTDs came online at once.

It wasn't until I was back at my hotel that evening, when I was setting up the dataparsing routines to publish the new data formats on our web site, that I noticed that the Shallow CTD had stopped reporting within about an hour of startup.  At first I worried that I had done some kind of damage to the instrument's connectors, either up top or underwater at the instrument, to interrupt communications.  But nine hours after this failure the Deep CTD also went offline.  We've had many problems with this make of CTD (formerly Falmouth, now RDI-Teledyne) but this type of failure mode for two instruments suggests a programming error or a communications equipment failure.

In this case it's unlikely to have been a programming error since the program deployed to St. Croix a few weeks ago is nearly line-for-line the same as this one.  This leaves the possibility of a failure in the datalogger itself, or perhaps the SIO4 serial-port unit.  I would say a datalogger failure is unlikely as it should have had more immediate and far-reaching effects than just these two CTDs.  An SIO4 failure is a great deal more likely but this particular SIO4 was also communicating with the WXT (Vaisala "weather transmitter") and the "groundtruth" CT, and both of those (at least on Thursday) appeared to be functioning normally.

So in the case of the CTDs, it could be logger/SIO4 failure, cable failure, instrument failure, or some obscure programming error that for whatever reason doesn't manifest itself at St. Croix.  I should also mention that all of this station's underwater cables were replaced last year so they are less likely to be at fault.

Moving the "deep" sensors:

Back in April UPR's Dr. Roy Armstrong reported a higher degree of biofouling on the "deep" light sensor (BIC) compared to the shallow BIC.  He asked if we would consider moving the deep sensors (because we all agreed that the deep BIC and CTD should be more or less co-located) shallower.  I picked up this discussion last week and looked at the numbers.  Roy had suggested 3m shallower but I found only 3.07m separation between the two sets of CTDs so I replied with my own suggestion that we leave at least 1m of separation between deep and shallow instruments.  After some more back and forth with AOML scientists we all agreed on a 1m - 2m separation between sensors.

Rachel and I ran into problems on Tuesday, however, when we tried to move the platform higher.  Eventually we realized that the "collar" of the platform that hugs the station tube was now broken on one side.  It was built with two stainless steel allthread rods extending several feet out of the platform side (see photo below).  The back-collar piece slides over these rods and the whole thing is secured with washers and nuts in back.  But one of the allthread rods appears to have snapped off during removal.  [I'm not certain how old this mounting platform is.  I know we upgraded the shallow BIC's mount in 2010, replacing the original bare-allthread design, and that at the time the deep BIC's platform was already in place.  So it dates from sometime between 2006 and 2010.]

Click on photo to see a larger version.

I spent part of Wednesday trying to come up with a way that we could reuse this mount securely without the allthread rod.  I thought that wrapping a long hose clamp around the one side of the platform would be strong enough, but the 48" hose clamps seemed to be just a little too short to allow this.  Chaining together two 48" clamps would have been way too long (the notched part of the clamp allows only for about 10" of tightening).  I also tried chaining together two clamps of different lengths but the 48" clamps were slightly wider than the shorter clamps and could not be mixed/matched.  In the end I found that one 48" clamp could be forced around the collar with less than an inch to spare.  What I did not realize until Thursday was that the collar does not hug the pylon tightly enough for its two sides to meet, but leaves about an inch of allthread open to the ocean (c.f. the fouling pattern in this second picture, below).

Click on photo to see a larger version.

Back in the water on Thursday, we quickly realized that the hose clamps weren't going to be long enough.  We tried securing the platform with four-foot plastic zip ties but those clearly weren't going to stand up to even the gentlest of currents.  In the end we gave up on the mounting platform and simply hose-clamped the light sensor directly to the pylon (see photo, below, which shows the final configuration of the deep BIC and CTD).

Click on photo to see a larger version.

The parts of the mounting platform were brought back to shore and left in storage beside the CREWS box.  This is the large plastic red box in the shed by the dock (not the diving dock, but the other one) on Isla Magueyes, where we keep the climbing rungs, harness, and other supplies used during our visits.

Miscommunication with UPR:  For this trip I was looking forward to working for the first time with Diana Marcela Beltrán, who was our designated UPR/CREWS liaison.  This was a role she inherited from Wess Merton at some point in the last year or two.  As far as I'm aware this is purely a voluntary role, usually filled at UPR by grad students, and AOML to my knowledge does not pay UPR to provide this support.  So I try not to be too demanding.

After our last trip to St. Croix was completed (July 29th - August 2nd) I contacted Diana on August 5th to ask what her schedule would be like for the rest of the month.  At the time I was ccing Francisco Pagán, who had been our main UPR contact person since the station first went live in January of 2006, but later I learned that Francisco's contract was now ending and he was planning to leave UPR.  I didn't see Francisco during our week at UPR although I don't know whether this was because he'd already left or just because we were all so busy with our work.

Diana replied to my followup message (August 8th), saying she was at that time on vacation in Colombia but she would be back on August 26th to work with us, so we arranged our travel plans accordingly.  Our plans would call for working with Diana on the 27th and 29th (Tuesday and Thursday).

After we checked into our hotel on the evening of Monday the 26th I sent an email to Diana to coordinate a meet-up time for the next morning.  Late that same evening, however, Diana replied by email that her return flight from Colombia had been changed "until Friday" and she would be unable to work with us.  She had made arrangements "last week" to have us work in the field with Nibo instead.  As far as I know she had not told anyone at AOML about these changes in plans the week before.

Working with Nibo was good news in that I had worked with him once last year, and he was therefore familiar with how we needed the boat to be positioned/anchored.  But the last time I'd been traveling with an AOML diver who happened to be fluent in Spanish, and this time neither Rachel nor I could come up with more than a few words of Spanish between us.  And Nibo it seemed could not speak or understand English.

This was not generally a problem for our surface/aerial work, because I could communicate to Rachel what was going on and Nibo could step forward and assist Rachel as needed by watching what she was doing.  But the language barrier presented more difficulties when diving because Rachel and I were both in the water and language was our only tool for communicating our needs to Nibo in the boat.  We'd described generally what we'd planned before we started our dive, but when the BIC mounting platform failed we had to change the order of our work and we had no way of telling Nibo what to do.  Later (for example) we needed a pair of pliers, and on Thursday I needed another hose clamp, which I knew to be alone in a ziploc bag in one of my red boxes on the boat but I had no way of guiding Nibo to find what I needed.  Eventually we muddled through with simple words and gestures but at several points we decided to simplify our plans underwater rather than attempt to communicate more complex concepts across our language barrier.  I should mention that this is in no way Nibo's fault, because he was patient and helpful and friendly to the degree that we were able to communicate with him.  But this is something that should be raised as a red flag for future visits.

There was one moment on Thursday night when I was debating whether to try rescheduling our flights home and see if UPR could bring us back out to the station one more time.  Oddly enough, as the evidence mounted about more things going wrong (the second CTD failed at about 11pm local time), it became less urgent to return to the station on this visit simply because there was less and less likelihood that we could fix what was wrong.  But also weighing in to my decision was the fact that, without Diana or Francisco around, I had nobody at UPR that I could contact Thursday evening to ask about an unplanned Friday boat trip.  We thought maybe we could just show up at the university Friday morning and ask around but the likelihood of success of that approach was not great.  This was not a language-barrier problem so much as it was a void of support, with Diana and Francisco unavailable but nobody nominated to take their place.

Summary:  The station is offline and probably without power, although it may have returned to its only-during-the-day solar-panel-fueled activity.  As best as I can tell the two CTDs are not communicating with the datalogger and there's a good chance that they aren't even logging data to their own flash memory storage.  The deep BIC platform is broken and the sensor was left hose-clamped directly to the station.

I am short on ideas about what to try next, but I think at a minimum we should replace the entire brain unit with all-new equipment on our next visit.  Even that may not be enough if the permanently-installed solar panels are faulty, and I do not believe the solar panels can be removed and/or replaced without taking the entire station out of the water for repair on land.  I also think we should wait until hurricane season has ended before scheduling our next trip to this station.

This update was written by Mike Jankulak, with photos taken by Rachel Kotkowski.

La Parguera CREWS Station cleaning

[From an email from Diana Beltran, May 16, 2013 3:18 PM EDT]

On April 25/13, I made sensors cleaning station. We started at 11:00 am, depth sensor has a broken part, as a small bite. The sensors were all covered with filamentous algae and some white calcareous algae. The chains and the central column were partially covered with Bivalves, sponges, filamentous algae, some corals, etc.

Last Tuesday May 14/13, we made complete cleaning station. We started connecting the CTD at 7:55 am, then start cleaning at 10:09 am (sensors, center pole, and chains). We completed cleaning at 10:55 am. Then at 11:25 am we picked up the CTD. The sensors were all covered with filamentous algae. The chains and the central column were covered with Bivalves, sponges, filamentous algae, some corals, etc. They need urgent cleaning larger soon.

Diana Marcela Beltrán R
(posted by Mike Jankulak)

April - sensors cleaning

[From an email from Diana Beltran, April 25, 2013 2:19 PM EDT]

Today Thursday April 25/13, I made sensors cleaning station. We started at 11:00 am, depth sensor has a broken part, as a small bite. The sensors were all covered with filamentous algae and some white calcareous algae. The chains and the central column were partially covered with Bivalves, sponges, filamentous algae, some corals, etc.

Diana Marcela Beltrán R
(posted by Mike Jankulak)

April complete cleaning

[From an email from Diana Beltran, April 2, 2013 2:35 PM EDT]

Today, Tuesday April 2/13, we made complete cleaning station. We started connecting the CTD at 7:50 am, then start cleaning at 9:15 am (sensors, center pole, and chains). We completed cleaning at 9:55 am. Then at 11:15 am we picked up the CTD.

The sensors were all covered with filamentous algae. The chains and the central column were partially covered with Bivalves, sponges, filamentous algae, some corals, etc.

Diana Marcela Beltrán R
(posted by Mike Jankulak)

La Parguera CREWS station offline

I'm sorry to report that the CREWS station in La Parguera is offline again.  It seems to have been suffering a very slow power drain ever since our last visit in January (see attached chart, which shows data from Dec 1st through this past weekend, with 2013 days-of-the-year as the X-axis -- the blue lines drop to zero twice during the week we were visiting and cutting power temporarily to the station).

Click this image to see a larger version.

Possible causes?  It could be as simple as a loose connection in the brain wiring up top, or that the connector for the solar panels is loose.  In fact I would guess that that's the most probable cause.  Other possible causes, off the top of my head, could include a bad charger-controller, bad solar panels, or bad batteries.  But I would say it's almost impossible that there could be anything wrong with the batteries, and rather unlikely that the solar panels themselves could be at fault.

Unlike the other times this station has lost power, it is extremely unlikely that any of the underwater instruments have flooded.  That kind of short-circuit has very quick and dramatic effects, taking the station from 100% normal to completely offline in a matter of hours.  This problem, whatever it is, has been going on for months.

The solution?  There is nothing that can be done to diagnose/fix this problem that does not involve climbing and opening the pylon.  This means that someone from AOML (probably me) will need to visit.  It might be a kind of tricky problem to resolve since it may take hours or days following any attempted fix to know whether the fix was effective.

In the meantime, I must assume that the station will continue its slow power decline.  It seems like the transmitter no longer has enough power to transmit, but the datalogger can be expected to continue running (and logging data locally) for a while longer.  My guess is that if the present rate of power decline continues, the station could continue logging data locally for another two months.  [I would not expect the problem to resolve itself spontaneously, although it's possible it could.]

We will have to weigh our priorities and schedules here at AOML to determine how soon a fix can be attempted.

station cleaning

[From an email from Diana Beltran, March 11, 2013 4:06 EDT]

The last tuesday March 5/13, I made sensors cleaning station. We started at 8:30 am, the sensors were all covered with filamentous algae and some white calcareous algae. The chains and the central column were partially covered with Bivalves, sponges, filamentous algae, some corals, etc. 

Biological Survey and Transmitter Repair at the La Parguera CREWS Station

[Click on any photo in this message to see the full-sized version.  All photos taken by P. Fletcher and M. Jankulak.]

In the week of January 28th to February 1st, 2013, a team from AOML traveled to La Parguera, Puerto Rico, for operations on and around the Coral Reef Early Warning System (CREWS) station at Media Luna reef.  This is the station designated LPRP4 by the National Data Buoy Center and known informally as LPPR1 among the Coral Health and Monitoring Program (CHAMP) team members.

Travelling from Miami were Pamela Fletcher (AOML/UF/SeaGrant) and Mike Jankulak (AOML/UM/CIMAS).  This same team had visited La Parguera in August of 2012 for a successful equipment swapout but by mid-October the station's hourly satellite transmissions had mostly gone silent and its GOES transmitter and antenna appeared to be in need of replacement.  By all indications the station was continuing to operate normally in all other ways (and write its data locally to a memory card) except for the integrated Weather Transmitter (WXT) whose acoustic wind sensors had failed.  Thus the WXT was targeted for replacement on this trip as well.

Pamela had brought along a PVC frame (see photo) which she had designed and built in collaboration with AOML's Mike Shoemaker.  The base of the frame is a 1-meter box with 10-centimeter stripes, and the top holds an opening for the CHAMP underwater camera.

Assembling the camera frame

While at the station Pamela led two dives to lay out 25-meter transect lines with measuring tape and then follow along with the camera/frame, meter by meter, to make a permanent record of the biology and substrate types surrounding the station.  Each photo shows the 1-meter box as a size reference (see sample photo).  Four such transects were completed, one in each of the four cardinal directions starting from the base of the station.

A sample from the photo-transects

Meanwhile, Mike conducted several pylon climbs to replace the transmitter (Tuesday, see photo) and the WXT (Wednesday, see photo).

Removing the old satellite antenna

Installing the new WXT

Locally-stored data were downloaded using a laptop on the boat (see photo) and an updated datalogger program was uploaded to the station.

Downloading data to the field laptop

While at the station on Wednesday, two very large birds believed to be blue-footed boobies were seen resting on the extended structures and instruments of the station, including the analog anemometer, the surface light sensor, and the satellite antenna (see photo).

Bird action while work proceeds up top

These birds did not fly away even with Mike at the top of the station, and their guano thickly coats every surface and, in some cases, acts as a kind of cement making it difficult or impossible to extract the removable masts.

In the end this operation was largely successful.  The biological photo-survey of the area will act as a baseline for future studies of the station's environment.  Transmissions resumed after equipment replacement, although there continue to be short, intermittent outages.  With most possible sources of equipment problems eliminated we now think that the persistent presence of these very large boobies may be affecting the transmission signal strength.  The transmission success rate has risen to about 80% - 90%, which falls short of the program's (somewhat arbitrary) goal of 90% or greater, but is an undeniable improvement over the 0% success rate seen since last November.

Data from August through January have been recovered and are available from CHAMP personnel.  Data feeds of the near real-time data have resumed to the CHAMP web reports, to NDBC, to the G2 Ecoforecasting System, and to CHAMP's mySQL database.  In the coming month a collaboration between CHAMP and HRD's Sonia Otero is expected to result in a web-based query prototype for data from this database.

The AOML team is grateful to UPRM and the facility on Magueyes for its welcome and support, specifically to Francisco Pagan for overseeing all UPRM arrangements and logistics, to Milton Carlo for supporting these dive operations, and to Chico, Wess and Orlando for their expert handling and anchoring of boats next to the station.

Monday November 26 María, Aníbal and I made almost!! complete cleaning station (three chains were missing for cleaning). We started cleaning up at 7:30 - 9:30 am (sensors) and part of center pole. Then at 9:30 am I connected the CTD, but we had to keep working on cleaning the station due it was very dirty (cleaning chains and finish the central pole). We completed cleaning at 12md.  I picked up the CTD at 2:30 pm.

The sensors were covered with green, red encrusting algae and filamentous algae. The chains and the central column were totally covered in Bivalves, sponges, corals, etc.  Our next visit is scheduled for the week of 10 to 14 December.

Basic cleaning done on October 22 / 2012, cleaning the sensors and wiring review, everything is properly connected, any cable is not broken. Cleaning of the center column and some of the chains. Cleaning was from 8 am to 10:30 am.

data feeds coming back online

The CHAMP web server has recovered to the point where the "near real time" data reports are once again online.  These reports are updated once per hour, and contain 12 hours' worth of data:

• Hourly Summary
• Hourly Detail
• Hourly Winds (6-minute data)

These reports are updated once per day, and contain 3 days' worth of data:

• Daily Summary
• Daily Detail
• Daily Winds (6-minute data)

Also, the feed of LPPR1 data to the National Data Buoy Center has resumed.  NDBC knows this station under the name "LPRP4" and its home page for the station is here:

•  NDBC's LPRP4 Station

NDBC's site has various views and graphs of recent data, and also some archives of older data.

The station has been running nearly 3 weeks since its overhaul, and for the most part the news is good.  The new batteries appear to be fully charged and maintain voltage levels a bit higher than seen with the old batteries.  All instruments continue to communicate with the logger.  The two SeaBird CTDs are showing an agreement on salinity measurements that was rarely found with the Teledyne CTDs we usually use, which raises hopes for better CTD performance this year.

The GOES transmissions were spotty at first but as of this writing we are receiving 100% of the expected transmissions for over a week.  Initial (post-overhaul) transmitter performance was running at only 50% success rates for a period of many days.  It's not clear why this should be so, and why performance has so substantially improved.  One possibility is that the satellite antenna (which was not replaced on this trip), whose underside was showing signs of peeling paint, has had some kind of moisture damage; or that the satellite antenna cable was damaged.  This wouldn't explain why transmissions improved, however.  Another possibility that occurs to me is perhaps the transmitter antenna is partially blocked... by birds.  We see lots of boobies resting on the aerial instruments at this station, but it's not known if they could "block" the satellite transmissions by sitting on the antenna.  We may want to device an antenna bird-deterrent for this site.

If there's bad news, then it's related to the secondary measurements of winds by the "WXT" Weather Transmitter made by Vaisala.  On this trip we unexpectedly found that the custom-made bird deterrent on the old WXT had been overcome by the boobies.  Five of the six spikes were broken off and the sixth was bent sideways.  That meant we had no deterrent for the new WXT, and indeed we are already seeing the unrealistically-high wind signals that suggest the sonic transducers are being blocked by birds.  However, we still have wind data from the anemometer, and the WXT will continue to report other measurements (air temperature and pressure, humidity).  The WXT's precipitation reports may also be unreliable.

But all in all the news is good!

Mike J+

2012 swapout: aftermath (5 of 5)

The AOML team consisting of Pamela Fletcher (UF/SeaGrant) and Mike Jankulak (UM/CIMAS) returned to Fort Lauderdale on Friday, August 31st, 2012.

All evidence so far indicates that the swapout operation was successful in its goals:  to swap out all meteorological and oceanographic instruments, to replace all underwater cables, to install new rechargeable batteries, to deploy SeaBird Microcat CTDs and their new programming.  Data obtained from the in-boat radio connection and the GOES satellite transmissions suggest that all of these sensors are operational, and have remained so during the station's first four days of uptime.

One possible area of concern is the station's satellite transmitter.  Since the station was not operational for the preceding four months, there was no recent data on how well its satellite transmitter (and its associated antennae) were working.  The station has for quite some time shown a pattern of dropping transmissions chiefly in the nighttime hours, and it was believed that this was caused by failing batteries (and should therefore now be resolved).

However, the station's first four days of operation have shown decidedly mixed transmission results.  These are:

• August 31st:  23 of 24 transmissions received
• September 1st:  13 of 24 transmissions received
• September 2nd:  13 of 24 transmissions received
• September 3rd:  23 of 23 (so far) transmissions received

These numbers suggest that there may be something wrong with the station's GOES transmitter, and that this condition may be intermittent.  More time will be needed to observe the station's transmitter performance and determine whether a particular fault can be identified.  One possible explanation is the previously-noted (mentioned here, with photo) observation of the transmitter's satellite antenna with its peeling paint and the possible implications for moisture intrusion.  Another factor may be that the new batteries have not yet been fully charged by the solar panels, although they achieve higher voltages with each passing day.

In any case, problems with the transmitter should not affect station sensors and data collection which appear, as of this writing, to be fully operational.

2012 swapout: reinstallation (4 of 5)

[As always, click photos to see their larger versions.]

On a happy note, I woke up early this morning with my head spinning with ideas about the two CTDs that would not communicate, and the idea came to me that perhaps their "instrument transmit" and "instrument receive" wires (Tx and Rx) had been reversed in our specially-made pigtail adapters.  If that was the case then I'd surely had to use "null modems" to connect to them in my office in January but perhaps I'd done just that and hadn't remembered it, due to the long delay between testing and deployment.  So when we arrived at the lab, the first thing I tried was to connect to the CTDs via null modems and hurrah!, that was successful.  Fortunately I had two null modems with me that I could wire into the brain connections at the top of the station later that day.

With that short delay we were still ready to leave the dock by 9am.  Upon arriving at the station I connected the five underwater instruments, on the boat, to the five new cables with cable-tie markings as follows:

1. "Groundtruth" CT, north side
2. Deep Light Sensor, south side
3. Deep CTD, south side
4. Shallow Light Sensor, north side
5. Shallow CTD, north side 

By 10am we were ready for our first dive, to install the instruments on the station.  The "groundtruth" CT would be temporarily cable-tied in place (see photo at right, taken before cables were tied down).  Once it was proven to be communicating correctly, it would be disconnected and its dummy-plugged cable end would be cable-tied loosely just below where the chains attached to the pylon near the ocean surface.

The CTDs were screwed down to two mounting brackets each, and each mounting bracket would eventually be hose-clamped to the pylon in two places.  During initial installation we used only two hose clamps per CTD instead of four (see photo of the deep CTD at left) in case we later had to retrieve the CTDs for additional work on the boat.

During this dive we also removed the last two of the old cables, for these had been tangled or cable-tied and we'd been unable to recover them on Tuesday on snorkel alone.  We also did a brief swim around the periphery of the station, going clockwise from pin to pin around the station.  In all this dive took 49 minutes.

From about 11am to 3pm I climbed the station and installed everything that had been removed on Tuesday, but in reverse:  the new battery packs, the three aerial masts, and the "brain."  Above is a picture of the inside of the "brain" chamber at the top of the pylon before reinstalling the batteries; right is a picture of me installing the Vaisala WXT.  Before installing the brain I first had to wire six plugs (four wires each) to the ends of the five new underwater cables and the new cable I was using for the surface light sensor.  After the brain was installed and everything was connected, I flipped the power switch and turned it on.

Next I returned to the boat to connect from my laptop (via radio) to the brain to ensure that all instruments were working properly.  However, I could get no connection at all and I had not remembered noticing any lights on the brain when I turned it on, so I decided to climb again and check if any obvious connections were unplugged.  I did so, powering off the brain, checking connections, and then carefully watching the lights when powering it back on.  I could see the serial ports briefly light up red on startup, and the radio unit flashing red to indicate that it was waiting for a connection, so I returned to the boat and tried again.  When there was still no connection, I switched radios on the boat and tried a different unit, and this time I was able to connect.  At this point all instruments were reporting correctly except, alas, the two SeaBird CTDs.  However I knew that the datalogger programming was written to set up the CTDs to report every six minutes, and at several points it had to wait until the exact six-minute mark before proceeding, so it might take six, twelve or even eighteen minutes before the CTDs would start reporting.  I returned to the pylon to plug up its external openings and tie down all loose cables, in preparation for closing it up, while we waited to see if the CTDs would wake up on their own.  When I next checked the CTDs they were fine, so I finished closing up the station top and removed the safety line and rungs.

One further note about the station top:  the bottom of the transmitter's satellite antenna (see photo at right) is showing signs of peeling paint and might be suffering moisture damage.  We should keep a careful eye on the transmission success rate and perhaps replace this antenna on our next trip regardless of whether it appears to be working normally or not.

At 4:30pm Pamela and I began our final dive to retrieve the groundtruth CT and lock down the other four instruments, and their cables, more securely.  The deep light sensor (whose platform is of a different design than that used for the shallow light sensor) was sitting loosely in its mount so we attempted to gum up the loose space with aquaseal.  A note for future deployments: normally I don't use mastic tape at the point where the light sensors are encircled by their mounts but for this station's deep light sensor we should make an exception.

Pamela noted where the station's lone "warning" buoy (originally there were four, now only one remains) has fouled considerably and its chain is covered with barnacles that appear to pose a chafing danger to the spectra line on that support chain (see photo at right).  We would suggest removing or replacing this buoy.

I will also include a photo (at left) of the bottom portion of the station, to show how much bio-fouling there has been since these spectra line were replaced in March.

A final underwater photo (at right) illustrates very well how completely fouled the chains are.  You mostly cannot see where one link ends and another link begins, and in most places you could not stick your finger through the hole in these chains (much less determine whether they still have sacrificial zincs installed).  Our time at the station was severely limited on this trip and we did not have any time for cleaning the chains or spectra, but this type of cleaning should be done with very high priority once it is determined who will be doing the monthly cleanings from here on out.

We took some post-swapout photos of the station (like this one, below) and then returned to the dock by 5:30pm, where we spent the next two hours cleaning our gear and packing our instruments and supplies for their return shipment to Miami.

[Photos taken by Pamela Fletcher and Mike Jankulak.]

2012 swapout: land-based work (3 of 5)

The agenda for Wednesday was to prepare the underwater instruments for deployment and then load up the boat and return to the station.  After installing the underwater instruments, we could then conduct an informal "reef check" exercise to be led by Pamela in the vicinity of the station.  We had decided not to follow strict reef check protocols because of the unsuitable topology of the ocean floor in that area (on this trip we were both limited to a maximum depth of 30 feet), but rather to "rehearse" reef check methods for future use and at the same time to collect some notes about the biological environment of the station.  There was no plan to do any pylon climbing on this day, instead we would return to the lab after diving so that I could reprogram/rewire the instruments so that we would be ready for Thursday's reinstallation and activation of the station.

Prepping the instruments mostly involved installing the batteries in the SeaBird CTDs and connecting to them to set their dates and times and verify that they were working.  We had planned to leave the dock by 9:30am but unfortunately the CTDs would not respond to any communication efforts, either one of them, and by 10am I decided to cancel the day's boat work and diving to devote myself to the electronics entirely.

By noon I had consulted with SeaBird technical support and what AOML personnel could be reached, and I was still unsuccessful in communicating with the CTDs.  [Note:  tech support's first instruction to me was to connect to the SeaBird CTDs with their own SeaBird communications cable, which to my knowledge we do not have.  If we continue to deploy these SeaBird CTDs we should definitely acquire and travel with one of these cables for use in debugging problems like these.]

I then turned to the other electronics work:  downloading the station's local memory files, uploading the new datalogger program, reprogramming the serial port units (SIO4s), replacing the four aerial instruments on their three masts (the anemometer and electronic compass share the same mast), replacing old cables, and testing all plugs to ensure that their wire/pin connections were unbroken.  I also cut away the duct tape from the old battery packs and taped up the new batteries with spectra-line handles in the same way.

During this work, one of the "brain" fuses was discovered to have blown, and was replaced.  It's not clear when this happened.

While working away in the lab I managed to meet up with Wess Merten, who has been doing the monthly station cleanings for several years, and asked him for three things:  one, the "groundtruth" CT sensor which I needed to connect on Thursday to ensure that its cable and wiring were properly connected; two, the deep light sensor that had been removed in July of 2011; and three, the shallow CTD that had been removed in March of 2012.  He was able to locate the first two sensors immediately but he did not know the exact whereabouts of the CTD and said that he would look for it.

By the end of the day everything was ready for deployment but the CTDs were still non-operational.  I wondered if my new laptop's USB-to-Serial adapter was somehow failing to communicate with the instruments, although it worked fine with the light sensors and the datalogger.  [Also, I connected to the shallow light sensor that we'd retrieved the day before and it still appeared to be operational.]  There was no good explanation for why (1) both CTDs were unable to communicate when (2) they had both been operating normally during testing in my office last January.  I decided we could try deploying them anyhow, reasoning that if the USB-to-Serial adapter was somehow to blame then perhaps the direct connection to our datalogger would work as it had in January.  In the worst case, if neither CTD would work, we could permanently install the "groundtruth" CT until such time as replacement CTDs could be installed.

2012 swapout: equipment removal (2 of 5)

[As always, click photos to see their larger versions.]

Tuesday morning, we got out on the water shortly after 10am, and Anibal Santiago Mercado ("Nibo") maneuvered the boat so that it was anchored into the current with the stern hanging back exactly level with the pylon.  Throughout Tuesday's and Thursday's work it was extremely easy to step from boat to pylon and back again, even despite some moderate wind and waves.  We started with a quick pylon climb to shut off the station's power so that the underwater instruments and cables could be removed without worrying about whether the cable plugs needed to be kept out of the water.

As mentioned previously here, two of the station's four underwater instruments had already failed and been removed, so there remained only two more instruments to remove today:  the shallow light sensor and the deep CTD.  As soon as we saw the deep CTD (see right), it was obvious why the station was without power -- the CTD's plug was snapped off and hanging at a 45-degree angle, with its internal wiring clearly visible.  It's impossible to say for sure whether this happened back in April when the station last lost power, or whether something else failed in April and this CTD connector broke more recently, but it seems reasonable to assume that this broken plug was the cause of the April power failure.  Note too that this CTD did not come back online at the point in time that the station recovered from its February 27th - March 31st downtime, so its communication wiring may have been impacted as early as late February.

The shallow light sensor (see photo at right) was completely covered with growth, including its mounting platform, its connector plug, and its sensing surface.  The last monthly cleaning recorded in this blog was in October of 2011, but there may have been more recent cleanings than this.  This particular location tends to foul more quickly than any other CREWS station and, given that the station was believed to be completely offline beginning in mid-April (and that its instruments would be completely replaced during this swapout operation that would bring it back online) there was no urgent need for instrument cleaning.

However, the station's support lines and chains are now quite considerably fouled (see photo at right), and should receive a thorough cleaning as soon as possible.

The morning's dive took about half an hour and by noon we were done.

Next, I climbed the pylon to disconnect and lower the "brain" (control unit), the three aerial masts with their four instruments (Vaisala integrated weather transmitter or "WXT," anemometer, electronic compass and surface light sensor), and the two battery packs.  As mentioned in yesterday's blog entry, these batteries had been continuously deployed since January of 2006 and in this trip we would be replacing them with new batteries of the same model.  The photo at left shows me lowering one of the two battery packs down to the boat.

One unpleasant surprise was that the Vaisala WXT's custom-made bird deterrent had not survived its maltreatment by the boobies that like to sit on every available surface of this station.  More so than at any other CREWS station, this station's surface instruments are returned to us with a very thick layer of bird guano covering everything.  Since we don't ship new WXT bird deterrents for each swapout (although henceforth, we will do so), and since only one of the deterrent's original six spikes remained (and even this spike was bent flat), this means that the new WXT will have to be deployed with no bird deterrent and its wind/rain measurements may therefore degrade more quickly than usual.

Another problem is that the station's grounding cables, used to electrically connect all the aluminum masts with the electrical ground of the electronics, the lightning diffuser brush, and the grounding plate mounted near the ocean floor, are now falling apart.  The spare grounding cables brought along on this trip were built with a different station design in mind and were too short to be useful for this station, but I repaired/replaced them as far as was possible.

Last on today's agenda was replacing the underwater cables.  This station is unique among CREWS stations in that its internal conduits (which run from just below the solar panels down to the ocean) have their openings beneath the ocean's surface.  This experimental design was not repeated elsewhere because those internal conduits quickly fill with barnacles and other sharp growth and this greatly impedes any effort to add, remove or replace the underwater cables on this station.

A survey of the underwater cables during our dive revealed that there were two cables emerging from the south conduit (which were used for the two deep instruments) and four cables in the north conduit.  This was one more cable than I recalled, but the fourth cable was originally installed for the SAMI pCO2 sensor which has not been present for many years, and the SAMI cable is a simpler (less tough) design compared to the "fishbite" cables used for all other instruments.  Since the SAMI cable was no longer being used and its polyurethane coating was cracking open where it had been exposed to the sunlight, we simply removed and discarded it during this visit.

The five new cables had been prepared by taping their ends (for when they would briefly be submerged during installation) and attaching a distinct number of cable ties (one through five) to each end so that the cable ends could be matched up and the correct cable could be wired to the correct instrument.  During this operation we ran cables #1, #4 and #5 up the northern conduit for the shallow instrument and cables #2 and #3 up the southern conduit for the deep instruments.

We started with the southern conduit.  We taped the end of a "fishtape" spool of hard plastic wire to the clipped ends of the deep instrument cables, and pulled them down the conduits from below.  Throughout this part of the work, I (Mike) stayed atop the pylon and Pamela frequently entered the water on snorkel to pull or push the cables past some obstruction or other.  Nibo stayed in the boat and taped the end of the new cables to the fishtape.  We were pleased and gratified when the two old cables pulled free of the station with almost zero resistance but met with difficulty when trying to pull the two new cables back up the conduit to replace them.  This was mostly due to debris and growth inside the underwater conduit.  After three tries, we eventually managed to feed a single cable up the conduit and then tie the second cable end to the middle of the first cable and feed both cables that way.

Turning to the northern conduit, there was one additional complication -- this conduit also held a green grounding wire (see photo at right, taken during Thursday's work) that ran from the mast/electronics/brush grounding system to the grounding plate at the base of the pylon.  Even if we'd wanted to try feeding all three new cables up to the top of the pylon (which we did not, given the frustration we'd had with trying to feed two cables at once up the other side), we did not want to pull out all four cables from the conduit at once for fear that they might be wrapped around the grounding wire and might cause it to pull loose or break.

So we decided to feed the new cables one by one.  We started by pulling out the SAMI cable, since it was distinguishable from the other three at both the top and the bottom, and fed a new cable to replace it.  Then it occurred to us that we could pull free a second cable at random without using it to feed a new cable, thereby freeing up some space in the conduit.  After that we replaced the last two cables one by one.  This took much longer than expected and we didn't leave the station that afternoon until almost 5:30pm.  But we were well-poised for the rest of the week's work.

[All photos taken by Pamela Fletcher.]

2012 swapout: preparations and travel (1 of 5)

On Monday, August 27th, 2012, a team from NOAA's Atlantic Oceanographic and Meteorological Laboratory (AOML) traveled to La Parguera, Lajas, Puerto Rico, in order to spend the week swapping out equipment on the Coral Reef Early Warning System (CREWS) station at Media Luna Reef.  The team consisted of Pamela Fletcher (University of Florida / SeaGrant / AOML) and Mike Jankulak (University of Miami / AOML).

Normally a CREWS station is visited once every year and its oceanographic and meteorological instruments are swapped out, its programming is updated, and its locally-stored data are recovered.  This trip was different from our usual swapouts in several important ways:

• The last swapout operation at this station was in November of 2010, nearly two years ago.  This current trip had been in the planning stages since January of 2012 but personnel shortages had made it impossible before now.  Therefore the station's instruments, cables and other hardware had had almost twice as long as usual to degrade and might be expected to be in worse condition than usual.
• Since the last swapout, this station had suffered three power failures.  The first failure (described here) was on July 14th, 2011, and was remotely diagnosed as a failure of the deep light sensor.  Fortunately the University of Puerto Rico was able to react quickly and the failed sensor was removed on July 15th, and station operations started coming back online two days later.
• The second failure was on February 27th, 2012, and this time the shallow CTD was indicated by the data record to be the most likely cause.  The UPR folks were too busy to visit the station until another team from AOML arrived in the week of March 12th - 16th to replace the station's support lines.  The shallow CTD was removed during these operations and the station regained power a bit more than two weeks later, on March 31st.  At the time that operations resumed, the deep CTD was found to be offline as well.
• Then after another two weeks, the station lost power a third time and never recovered.  The cause of this third power failure remained unknown up until the time of this current August/2012 swapout operation.  Since there had been this many problems with power failures, and since this station's electronics and cables were originally deployed more than six years ago, we made the decision to replace all of the station's underwater cables on this trip.
• In September of 2011, after the first of the three power failures described above, I conducted an analysis of the station's recent record of dropped satellite transmissions.  This analysis revealed that the station's "daylight" transmission success rate was about 86%, which was lower than we'd like but still not that bad, but its "dark" transmission success rate was only 57%, which was terrible.  This strongly suggested that the station was more likely to supply enough power to the transmitter while its solar panels were providing additional power to the system, whereas when powered by only the batteries there was often not sufficient power.  Given that the batteries were then halfway through their sixth year of deployment (and an additional year has now passed since that analysis took place), and given additionally that the batteries had been fully drained by more than one short-circuit event, we decided to replace the batteries on this current swapout trip.
• One final difference in this year's swapout was that we had elected to try a different make of CTD in this year's deployment.  Instead of deploying our usual RDI/Teledyne (formerly Falmouth) CTDs, this year we elected to deployed two refurbished SeaBird Microcat SMP37 CTDs.  These CTDs require different programming and their maintenance needs are different from the Teledyne CTDs.
• On a minor note, this swapout would also be the first such operation to be conducted with the new CREWS travel laptop, following the failure of the old laptop's battery supply.  The new laptop runs Windows 7 compared to the old laptop's Windows XP, and the old laptop had a built-in serial port whereas the new laptop requires the use of a USB-to-Serial adaptor.

Arrangements for this week of work began in July, when our replacement instruments and tools were shipped from Miami to UPR.  Air travel and hotel arrangements were finalized in early August.  Coordinating with UPR's Francisco Pagan, we arranged for boat and diving support, with boat operations to be carried out by Anibal Santiago Mercado, for the three days from Tuesday to Thursday.

There was some last-minute doubt about the timing of this trip with the formation of TS Isaac and its effects on UPRM (in the week before this operation) and Miami (in the days before we were scheduled to fly out of Fort Lauderdale).  In the end, however, the trip was able to go ahead as planned and we (Pamela and Mike) arrived in La Parguera Monday evening, ready for work to begin on Tuesday morning.

Complete Cleaning in October

Today (Monday, October 3rd, 2011), a complete cleaning (CTD validation, cleaned all sensors, chains, lines, and spar) was done at the CREWS Station in La Parguera, Puerto Rico, by Wessley Merten. The groundtruth validation sensor was placed on at 0945 and was removed at 1324.

Basic Cleaning in August

Yesterday, Tuesday August 30th, Wess Merten conducted a thorough basic cleaning and assessment of the CREWs station in La Parguera. I conducted a surface inspection, underwater inspection/cleaning, optical sensor cleaning, and a CTD sensor cleaning.  I also very carefully removed some biofouling that has accumulated on some of the cables.  In addition, all four brass filters were replaced on the CTD's.  After more than 3 hours underwater the station looks great!  A complete cleaning will be conducted next week.