complete cleaning

[From an email from Diana Beltran, November 18, 2013 5:02 PM EST.]

Last Wednesday November 13/13, we made complete cleaning station. We started connecting the CTD at 8:05 am, then start cleaning at 10:10 am (sensors, center pole, and chains). We completed cleaning at 10:55 am. Then at 1:40 pm we picked up the CTD.

The sensors were all covered with filamentous algae. The chains and the central column were partially covered with Bivalves, sponges, filamentous algae, some corals, etc.

Diana
(posted by Mike Jankulak)

CREWS La Parguera

[From an email from Diana Beltran, October 31, 2013 4:32 PM EDT.]

On October 8 and today 31 /2013, I made sensors cleaning station. We started at 8:00 am. The sensors were all covered with filamentous algae and some white calcareous algae. The chains and the central column were all covered with Bivalves, sponges, filamentous algae, some corals, etc. I tried to clean the chains but has been hard work, I cleaned three so far, so I will continue cleaning them.

Diana
(posted by Mike Jankulak)

post-mortem: what went wrong last week in Puerto Rico

During the week of August 26th - 30th, two members of the CHAMP team (Rachel Kotkowski and Mike Jankulak) visited the CREWS station at La Parguera, PR.  Our goals were:

• Identify and fix whatever had taken the station offline last March.
• Swap out all instruments, since it is now one year after last August's swapout.
• Move the "deep" instruments several metres higher.

Unfortunately we encountered problems from start to finish and fell short of nearly all of these goals, to wit:

• The station lost power again on August 31st, only two days after reinstallation.
• Both CTDs on the station failed the same day they were installed.
• The mounting platform for the deep BIC (light sensor) is broken and the sensor is currently hose-clamped directly to the pylon.
• Our UPR CREWS contact was unexpectedly absent and we had to work across a language barrier with her replacement.

Arguably the station is worse off compared to how we found it at the beginning of the week, and we still have no clear idea of what the problems are.  Below I will go into more detail about each of the issues listed above.

First, the power failure.  I had last visited the station in January and in the weeks following that visit I noticed that it seemed to be running off of battery power only, with no daily recharge from the solar panels.  At the time I guessed maybe that the solar panel connector-plug had not been fully plugged in at the top of the pylon, or perhaps one of the connector's pins had broken off.  Last Tuesday I recovered the "brain" (control package) and both sets of batteries from the station and brought them back to land.  The batteries I found to be drained to about 3.5V (compared to a normal range from 12V to 14V), and I connected them to a power supply for a quick charge back to near-normal levels.

However, I could find no evidence of loose pins, broken wires, or failed connectors, and both fuses in the brain were intact.  Far more confusing was the evidence in the logger's memory cache of what had been going on after the transmissions failed in mid-March.  As I'd thought, the station from January to March appeared to be slowly draining its batteries until levels were too low for transmitting.  But after a few weeks of no activity at all, the station in early April appeared (by evidence of its power levels) to have developed the opposite problem from before.  Instead of batteries connected without solar panels, from April to August the station appeared to have solar panels connected without batteries.  The new pattern of voltages began April 4th, which is two days after one of the monthly cleanings on April 2nd.  I do not know if there is any logical connection between these two events.

Click on graph to see a larger version.

Above is a graph of power levels since the beginning of 2013 (the x-axis is labeled with the day of year, from 0 to 365, and the y-axis is voltage).  There is a normal diurnal voltage pattern until my visit at the end of January, after which there are only a few days when battery charging occurs but mostly the power levels steadily decline.  In mid-March the power levels fall below 9V and all station activity ceases.  However, on April 4th the station power levels start fluctuating wildly between 0V at night to more than 17V in the daytime, which is a pattern I have seen when a station has lost its connection to its batteries but still has solar panels plugged in.  In this situation the station generally doesn't have a steady enough power supply to complete even one full transmission, although we did receive just one report on June 2nd (which led to a lot of head-scratching, since at that point we believed the station to be entirely without power).

As I've said, I could not find any "smoking gun" to explain these problems.  The evidence suggests that there has been intermittently faulty connections to the batteries and the solar panels, but not at the same time.  I'd brought a new charger-controller with me and, since this was one element that connects to both batteries and solar panels, I replaced it in the hopes that improvements would follow.  I also speculated that perhaps some of the brain hardware (such as the on/off switch) might be causing the problem.  The "brain" is made up of a backboard of fibreglass with many simple hardware elements screwed or soldered on permanently.  I have often replaced the main electronics components (datalogger, SIO4 serial-port units, satellite transmitter, RF radio) but I have never replaced the baseline hardware elements nor do I have the tools or expertise to do so.  To my knowledge none of these simpler hardware elements have ever failed at a CREWS station before now.

In any case I reinstalled the brain on Thursday, knowing that it would likely take several days of charging to know whether the solar panels and batteries were working properly again.  In fact it took only two days for transmissions to fail this time, probably because the batteries had been drained for months and two days of charging them off my small power supply was not enough to keep them going very long.

I am unsure what might be required to fix the power problems now.  One conservative approach would be to build an entirely new "brain" package, including all switches, connectors, screw-down boards and fuses.  Even this, however, is not guaranteed to help if there is a wiring problem with the permanently-installed solar panels on the station.  Also the wiring of the solar panels at this station is different from all other CREWS stations past or present and replacing the brain entirely might require careful examination and testing of those solar panels (or in the worst case, replacement of the solar panels).

Next, the CTDs:  As usual I reinstalled all station equipment and powered it on, then returned to the boat to connect by radio from my laptop to the station to verify that everything was working properly.  The CTDs have a more complicated program than most instruments -- the datalogger essentially "programs" them to run in interval mode and sets their clock according to its GPS-derived time.  The CTDs when online will spontaneously offer up new readings every 6 minutes but it often takes 6, 12 or even 18 minutes after startup before they start reporting.  So I checked in every 6 minutes and after a few cycles both CTDs came online at once.

It wasn't until I was back at my hotel that evening, when I was setting up the dataparsing routines to publish the new data formats on our web site, that I noticed that the Shallow CTD had stopped reporting within about an hour of startup.  At first I worried that I had done some kind of damage to the instrument's connectors, either up top or underwater at the instrument, to interrupt communications.  But nine hours after this failure the Deep CTD also went offline.  We've had many problems with this make of CTD (formerly Falmouth, now RDI-Teledyne) but this type of failure mode for two instruments suggests a programming error or a communications equipment failure.

In this case it's unlikely to have been a programming error since the program deployed to St. Croix a few weeks ago is nearly line-for-line the same as this one.  This leaves the possibility of a failure in the datalogger itself, or perhaps the SIO4 serial-port unit.  I would say a datalogger failure is unlikely as it should have had more immediate and far-reaching effects than just these two CTDs.  An SIO4 failure is a great deal more likely but this particular SIO4 was also communicating with the WXT (Vaisala "weather transmitter") and the "groundtruth" CT, and both of those (at least on Thursday) appeared to be functioning normally.

So in the case of the CTDs, it could be logger/SIO4 failure, cable failure, instrument failure, or some obscure programming error that for whatever reason doesn't manifest itself at St. Croix.  I should also mention that all of this station's underwater cables were replaced last year so they are less likely to be at fault.

Moving the "deep" sensors:

Back in April UPR's Dr. Roy Armstrong reported a higher degree of biofouling on the "deep" light sensor (BIC) compared to the shallow BIC.  He asked if we would consider moving the deep sensors (because we all agreed that the deep BIC and CTD should be more or less co-located) shallower.  I picked up this discussion last week and looked at the numbers.  Roy had suggested 3m shallower but I found only 3.07m separation between the two sets of CTDs so I replied with my own suggestion that we leave at least 1m of separation between deep and shallow instruments.  After some more back and forth with AOML scientists we all agreed on a 1m - 2m separation between sensors.

Rachel and I ran into problems on Tuesday, however, when we tried to move the platform higher.  Eventually we realized that the "collar" of the platform that hugs the station tube was now broken on one side.  It was built with two stainless steel allthread rods extending several feet out of the platform side (see photo below).  The back-collar piece slides over these rods and the whole thing is secured with washers and nuts in back.  But one of the allthread rods appears to have snapped off during removal.  [I'm not certain how old this mounting platform is.  I know we upgraded the shallow BIC's mount in 2010, replacing the original bare-allthread design, and that at the time the deep BIC's platform was already in place.  So it dates from sometime between 2006 and 2010.]

Click on photo to see a larger version.

I spent part of Wednesday trying to come up with a way that we could reuse this mount securely without the allthread rod.  I thought that wrapping a long hose clamp around the one side of the platform would be strong enough, but the 48" hose clamps seemed to be just a little too short to allow this.  Chaining together two 48" clamps would have been way too long (the notched part of the clamp allows only for about 10" of tightening).  I also tried chaining together two clamps of different lengths but the 48" clamps were slightly wider than the shorter clamps and could not be mixed/matched.  In the end I found that one 48" clamp could be forced around the collar with less than an inch to spare.  What I did not realize until Thursday was that the collar does not hug the pylon tightly enough for its two sides to meet, but leaves about an inch of allthread open to the ocean (c.f. the fouling pattern in this second picture, below).

Click on photo to see a larger version.

Back in the water on Thursday, we quickly realized that the hose clamps weren't going to be long enough.  We tried securing the platform with four-foot plastic zip ties but those clearly weren't going to stand up to even the gentlest of currents.  In the end we gave up on the mounting platform and simply hose-clamped the light sensor directly to the pylon (see photo, below, which shows the final configuration of the deep BIC and CTD).

Click on photo to see a larger version.

The parts of the mounting platform were brought back to shore and left in storage beside the CREWS box.  This is the large plastic red box in the shed by the dock (not the diving dock, but the other one) on Isla Magueyes, where we keep the climbing rungs, harness, and other supplies used during our visits.

Miscommunication with UPR:  For this trip I was looking forward to working for the first time with Diana Marcela Beltrán, who was our designated UPR/CREWS liaison.  This was a role she inherited from Wess Merton at some point in the last year or two.  As far as I'm aware this is purely a voluntary role, usually filled at UPR by grad students, and AOML to my knowledge does not pay UPR to provide this support.  So I try not to be too demanding.

After our last trip to St. Croix was completed (July 29th - August 2nd) I contacted Diana on August 5th to ask what her schedule would be like for the rest of the month.  At the time I was ccing Francisco Pagán, who had been our main UPR contact person since the station first went live in January of 2006, but later I learned that Francisco's contract was now ending and he was planning to leave UPR.  I didn't see Francisco during our week at UPR although I don't know whether this was because he'd already left or just because we were all so busy with our work.

Diana replied to my followup message (August 8th), saying she was at that time on vacation in Colombia but she would be back on August 26th to work with us, so we arranged our travel plans accordingly.  Our plans would call for working with Diana on the 27th and 29th (Tuesday and Thursday).

After we checked into our hotel on the evening of Monday the 26th I sent an email to Diana to coordinate a meet-up time for the next morning.  Late that same evening, however, Diana replied by email that her return flight from Colombia had been changed "until Friday" and she would be unable to work with us.  She had made arrangements "last week" to have us work in the field with Nibo instead.  As far as I know she had not told anyone at AOML about these changes in plans the week before.

Working with Nibo was good news in that I had worked with him once last year, and he was therefore familiar with how we needed the boat to be positioned/anchored.  But the last time I'd been traveling with an AOML diver who happened to be fluent in Spanish, and this time neither Rachel nor I could come up with more than a few words of Spanish between us.  And Nibo it seemed could not speak or understand English.

This was not generally a problem for our surface/aerial work, because I could communicate to Rachel what was going on and Nibo could step forward and assist Rachel as needed by watching what she was doing.  But the language barrier presented more difficulties when diving because Rachel and I were both in the water and language was our only tool for communicating our needs to Nibo in the boat.  We'd described generally what we'd planned before we started our dive, but when the BIC mounting platform failed we had to change the order of our work and we had no way of telling Nibo what to do.  Later (for example) we needed a pair of pliers, and on Thursday I needed another hose clamp, which I knew to be alone in a ziploc bag in one of my red boxes on the boat but I had no way of guiding Nibo to find what I needed.  Eventually we muddled through with simple words and gestures but at several points we decided to simplify our plans underwater rather than attempt to communicate more complex concepts across our language barrier.  I should mention that this is in no way Nibo's fault, because he was patient and helpful and friendly to the degree that we were able to communicate with him.  But this is something that should be raised as a red flag for future visits.

There was one moment on Thursday night when I was debating whether to try rescheduling our flights home and see if UPR could bring us back out to the station one more time.  Oddly enough, as the evidence mounted about more things going wrong (the second CTD failed at about 11pm local time), it became less urgent to return to the station on this visit simply because there was less and less likelihood that we could fix what was wrong.  But also weighing in to my decision was the fact that, without Diana or Francisco around, I had nobody at UPR that I could contact Thursday evening to ask about an unplanned Friday boat trip.  We thought maybe we could just show up at the university Friday morning and ask around but the likelihood of success of that approach was not great.  This was not a language-barrier problem so much as it was a void of support, with Diana and Francisco unavailable but nobody nominated to take their place.

Summary:  The station is offline and probably without power, although it may have returned to its only-during-the-day solar-panel-fueled activity.  As best as I can tell the two CTDs are not communicating with the datalogger and there's a good chance that they aren't even logging data to their own flash memory storage.  The deep BIC platform is broken and the sensor was left hose-clamped directly to the station.

I am short on ideas about what to try next, but I think at a minimum we should replace the entire brain unit with all-new equipment on our next visit.  Even that may not be enough if the permanently-installed solar panels are faulty, and I do not believe the solar panels can be removed and/or replaced without taking the entire station out of the water for repair on land.  I also think we should wait until hurricane season has ended before scheduling our next trip to this station.

This update was written by Mike Jankulak, with photos taken by Rachel Kotkowski.

La Parguera CREWS Station cleaning

[From an email from Diana Beltran, May 16, 2013 3:18 PM EDT]

On April 25/13, I made sensors cleaning station. We started at 11:00 am, depth sensor has a broken part, as a small bite. The sensors were all covered with filamentous algae and some white calcareous algae. The chains and the central column were partially covered with Bivalves, sponges, filamentous algae, some corals, etc.

Last Tuesday May 14/13, we made complete cleaning station. We started connecting the CTD at 7:55 am, then start cleaning at 10:09 am (sensors, center pole, and chains). We completed cleaning at 10:55 am. Then at 11:25 am we picked up the CTD. The sensors were all covered with filamentous algae. The chains and the central column were covered with Bivalves, sponges, filamentous algae, some corals, etc. They need urgent cleaning larger soon.

Diana Marcela Beltrán R
(posted by Mike Jankulak)

April - sensors cleaning

[From an email from Diana Beltran, April 25, 2013 2:19 PM EDT]

Today Thursday April 25/13, I made sensors cleaning station. We started at 11:00 am, depth sensor has a broken part, as a small bite. The sensors were all covered with filamentous algae and some white calcareous algae. The chains and the central column were partially covered with Bivalves, sponges, filamentous algae, some corals, etc.

Diana Marcela Beltrán R
(posted by Mike Jankulak)

April complete cleaning

[From an email from Diana Beltran, April 2, 2013 2:35 PM EDT]

Today, Tuesday April 2/13, we made complete cleaning station. We started connecting the CTD at 7:50 am, then start cleaning at 9:15 am (sensors, center pole, and chains). We completed cleaning at 9:55 am. Then at 11:15 am we picked up the CTD.

The sensors were all covered with filamentous algae. The chains and the central column were partially covered with Bivalves, sponges, filamentous algae, some corals, etc.

Diana Marcela Beltrán R
(posted by Mike Jankulak)

La Parguera CREWS station offline

I'm sorry to report that the CREWS station in La Parguera is offline again.  It seems to have been suffering a very slow power drain ever since our last visit in January (see attached chart, which shows data from Dec 1st through this past weekend, with 2013 days-of-the-year as the X-axis -- the blue lines drop to zero twice during the week we were visiting and cutting power temporarily to the station).

Click this image to see a larger version.

Possible causes?  It could be as simple as a loose connection in the brain wiring up top, or that the connector for the solar panels is loose.  In fact I would guess that that's the most probable cause.  Other possible causes, off the top of my head, could include a bad charger-controller, bad solar panels, or bad batteries.  But I would say it's almost impossible that there could be anything wrong with the batteries, and rather unlikely that the solar panels themselves could be at fault.

Unlike the other times this station has lost power, it is extremely unlikely that any of the underwater instruments have flooded.  That kind of short-circuit has very quick and dramatic effects, taking the station from 100% normal to completely offline in a matter of hours.  This problem, whatever it is, has been going on for months.

The solution?  There is nothing that can be done to diagnose/fix this problem that does not involve climbing and opening the pylon.  This means that someone from AOML (probably me) will need to visit.  It might be a kind of tricky problem to resolve since it may take hours or days following any attempted fix to know whether the fix was effective.

In the meantime, I must assume that the station will continue its slow power decline.  It seems like the transmitter no longer has enough power to transmit, but the datalogger can be expected to continue running (and logging data locally) for a while longer.  My guess is that if the present rate of power decline continues, the station could continue logging data locally for another two months.  [I would not expect the problem to resolve itself spontaneously, although it's possible it could.]

We will have to weigh our priorities and schedules here at AOML to determine how soon a fix can be attempted.

station cleaning

[From an email from Diana Beltran, March 11, 2013 4:06 EDT]

The last tuesday March 5/13, I made sensors cleaning station. We started at 8:30 am, the sensors were all covered with filamentous algae and some white calcareous algae. The chains and the central column were partially covered with Bivalves, sponges, filamentous algae, some corals, etc. 

Biological Survey and Transmitter Repair at the La Parguera CREWS Station

[Click on any photo in this message to see the full-sized version.  All photos taken by P. Fletcher and M. Jankulak.]

In the week of January 28th to February 1st, 2013, a team from AOML traveled to La Parguera, Puerto Rico, for operations on and around the Coral Reef Early Warning System (CREWS) station at Media Luna reef.  This is the station designated LPRP4 by the National Data Buoy Center and known informally as LPPR1 among the Coral Health and Monitoring Program (CHAMP) team members.

Travelling from Miami were Pamela Fletcher (AOML/UF/SeaGrant) and Mike Jankulak (AOML/UM/CIMAS).  This same team had visited La Parguera in August of 2012 for a successful equipment swapout but by mid-October the station's hourly satellite transmissions had mostly gone silent and its GOES transmitter and antenna appeared to be in need of replacement.  By all indications the station was continuing to operate normally in all other ways (and write its data locally to a memory card) except for the integrated Weather Transmitter (WXT) whose acoustic wind sensors had failed.  Thus the WXT was targeted for replacement on this trip as well.

Pamela had brought along a PVC frame (see photo) which she had designed and built in collaboration with AOML's Mike Shoemaker.  The base of the frame is a 1-meter box with 10-centimeter stripes, and the top holds an opening for the CHAMP underwater camera.

Assembling the camera frame

While at the station Pamela led two dives to lay out 25-meter transect lines with measuring tape and then follow along with the camera/frame, meter by meter, to make a permanent record of the biology and substrate types surrounding the station.  Each photo shows the 1-meter box as a size reference (see sample photo).  Four such transects were completed, one in each of the four cardinal directions starting from the base of the station.

A sample from the photo-transects

Meanwhile, Mike conducted several pylon climbs to replace the transmitter (Tuesday, see photo) and the WXT (Wednesday, see photo).

Removing the old satellite antenna

Installing the new WXT

Locally-stored data were downloaded using a laptop on the boat (see photo) and an updated datalogger program was uploaded to the station.

Downloading data to the field laptop

While at the station on Wednesday, two very large birds believed to be blue-footed boobies were seen resting on the extended structures and instruments of the station, including the analog anemometer, the surface light sensor, and the satellite antenna (see photo).

Bird action while work proceeds up top

These birds did not fly away even with Mike at the top of the station, and their guano thickly coats every surface and, in some cases, acts as a kind of cement making it difficult or impossible to extract the removable masts.

In the end this operation was largely successful.  The biological photo-survey of the area will act as a baseline for future studies of the station's environment.  Transmissions resumed after equipment replacement, although there continue to be short, intermittent outages.  With most possible sources of equipment problems eliminated we now think that the persistent presence of these very large boobies may be affecting the transmission signal strength.  The transmission success rate has risen to about 80% - 90%, which falls short of the program's (somewhat arbitrary) goal of 90% or greater, but is an undeniable improvement over the 0% success rate seen since last November.

Data from August through January have been recovered and are available from CHAMP personnel.  Data feeds of the near real-time data have resumed to the CHAMP web reports, to NDBC, to the G2 Ecoforecasting System, and to CHAMP's mySQL database.  In the coming month a collaboration between CHAMP and HRD's Sonia Otero is expected to result in a web-based query prototype for data from this database.

The AOML team is grateful to UPRM and the facility on Magueyes for its welcome and support, specifically to Francisco Pagan for overseeing all UPRM arrangements and logistics, to Milton Carlo for supporting these dive operations, and to Chico, Wess and Orlando for their expert handling and anchoring of boats next to the station.