Transmitter trouble

27 July 2009

Photograph of a 12 way VCSEL array mounted on a PCB. The two guide pins for the MT-12 fibre connector and the pins for the electrical connections are visible.

Optical transmitters used in the SCT, Pixel and Liquid Argon (LAr) sub-detectors have been failing at an alarming rate, but for the silicon detectors, at least, things are finally looking up.

Out of 400 12-channel arrays in the SCT, approximately one per day was dying over the last year or so, a rate which SCT Data Links Convenor, Tony Weidberg describes as “totally unacceptable”. The rate was a little slower in the Pixel detector, which was losing around one of its 272 8-channel arrays each week, but this still represented a major problem. “Doing nothing was not an option,” says Tony.

Number of dead TX channels in the pixel detector versus time (2008).

Both sub-detectors use optical laser transmitters designed and built in the same factory. The Vertical Cavity Surface Emitting Lasers (VCSELs) transmit Trigger Timing and Control data and calibration commands from the electronics in USA15 to the front-end modules on the detectors. A VCSEL ‘fail’ in this circumstance means that it stops emitting light.

The problem first came to light in summer 2008, and since then over 100 units have been replaced with spares on the SCT alone.

“We also have an in-built redundancy system, so if a single channel dies we can use the data from a neighbouring channel,” explains Tony, “so we were still able to take very good cosmic ray data last year, despite these problems.”

For the long-term though, it was essential to quickly identify what was going wrong, so that the problem could be tackled.

“We gradually got more circumstantial evidence that [the failures] were due to electrostatic discharge,” reports Tony. High levels of electrostatic discharge (ESD) would destroy the components immediately, but Tony and colleagues found out that exposure to low levels of ESD over time could compromise the lifetime of the VCSELs, even though outwardly they appeared to perform perfectly until they died.

“The most direct way of proving this is expensive transmission electron microscopy,” says Tony, “but instead we did experiments with an ESD gun to zap [components] with low static pulses and test their lifetime.”

They found that the VCSELs could die after weeks of operation under these pressures. Tony admits that this doesn’t constitute conclusive scientific proof that this is what is happening to the VCSELs in the detector, but, “all the symptoms are shown, so we’re reasonably confident that the damage was due to ESD”.

Because the active volume of the lasers is so small, they can begin to melt when exposed to just a tiny bit of static. Very strict guidelines for handling these super-sensitive components during manufacturing must be adhered to: the operators need to be grounded, wearing special clothing, and standing/sitting on a grounded, conductive surface.

“But you have to allow for the fact that this is done by humans, and people make mistakes,” concedes Tony. In April this year, he visited the factory in Taiwan where the original VCSELs were manufactured and where a whole new batch of replacements was being made. He offered suggestions for small improvements they could make in working practice, such as installing an alarm that triggers if the operator isn’t properly earthed, but overall he was happy that the manufacture was proceeding with much more caution and attention to detail this time around.

All the VCSELs will be replaced with these new ones before beam. At the time of publishing, 25 per cent of the new Pixel and SCT devices have been installed, and the SCT redundancy scheme means that it is functioning at very close to 100 per cent efficiency now. “The remaining three quarters of the new devices will be installed by the end of the month, so we’ll be in good shape for LHC running,” says Tony.

Tobias Flick, who is responsible for the Pixel optical links, is hopeful that the VCSEL problems are now behind them: “I am happy to receive the new plug-ins, and to replace everything in time before the next beam starts,” he says. “And we cross of course the fingers, that this problem is eliminated now.”

Over in the LAr sub-detector, VCSELs are used as individual devices rather than arrays, with one VCSEL on each 128-channel Front End Board (FEB) to transmit all of its readout data.  After more than a year of reliable running, the system of 1524 FEBs began to experience failures at a rate of almost one per week. Although attempts by a dedicated Task Force to pinpoint the cause have so far been inconclusive, ESD at the same Taiwanese source is the leading suspect. 

John Parsons, responsible for the LAr FEBs, reports that, "While we continue efforts to conclusively identify the failure mechanism and to estimate the total number of compromised devices, we are aggressively developing backup options.”

Those backups include a close replica of the current device being prepared by Stefan Simion, member of the Task Force, as well as a design that would double the links in order to provide some redundancy. “The dual-link option is attractive, but would require doubling the number of optical fibres between USA15 and the LAr detectors,” John points out. “Finding sufficient space, particularly in the endcap cable chains, is a difficult issue that is being investigated in cooperation with ATLAS Technical Coordination."

A decision must be made soon, and production of a backup launched, in case it proves necessary to replace all VCSELs during the next shutdown, when the FEBs will again be accessible.




Ceri Perkins

ATLAS e-News