Ready for rejection!

28 June 2010

Webcam photo of the control room May 24th

As mentioned in the last e-news issue, it was very early on May 25 that the ATLAS High-Level Trigger (HLT) has started to actively select events keeping the more interesting ones and rejecting the uninteresting ones.

“The trigger is like a big sieve with different levels of granularity.” explains Monika Wielers, the convener of the electron and photon trigger group. “Up to now it had been sufficient to leave it to the hardware-based Level-1 trigger to sieve out the interesting events as the incoming collision rate was low enough. At luminosities above 1 to 2 x1029cm-2s-1 this is no longer true as the number of collisions with electron and photon candidates selected by the L1 trigger with the lowest requirement of electromagnetic energy (ET>2GeV) detected is getting too high. So we decided to switch on the next level of selections: the HLT. One of the main criteria to say we are ready for rejection was that the trigger is very efficiently selecting collisions containing 'good' electron and photon candidates found by the offline reconstruction. It is important to stress that getting ready to enable the HLT in rejection mode involved much work of a considerable number of people (in fact too numerous to all list here), on trigger operation, menus and data analyses". Finally D-day was approaching during the Whit Sunday weekend.”

The weekend started well with a nice long run on Friday night followed by more collisions on Saturday, Sunday and Monday. ”Every time we hoped ’Now, now, the peak luminosity exceeds the critical threshold. Let's switch on the HLT selection‘”, recalls Monika. “but ... no luck.“ Anna Sfyrla, who had just finished her weekly shift as menu expert on call, remembers: “We were 6 to 7 trigger experts in the control room on Monday May 24th, all anxious about what would happen next - we also had bets between us on what the peak luminosity would be - people guessed from 1.2 to1 to 2 x1029cm-2s-1 and I think in the end Frank Winklmeier had the closest guess.” The next collision period arrived but again the luminosity was not high enough and no HLT rejection was required. In the evening only the shift crew was left in the control room.

Just before going to bed Frank, the online on-call expert during that weekend, thought: ”It would be ironic if we had to enable the rejection at night without any expert around in the control room...” Ironic or not, just a few hours later Frank was woken up by a phone call from John Morris, the trigger shifter on duty. The luminosity had reached 1 to 2 x1029cm-2s-1 and too many collisions flooded the system.

The situation was quickly under control after putting the correct set of triggers including the HLT selections in place. Frank then called Takanori Kono, the e/gamma trigger expert on-call, to check the performance. After having checked all the monitoring plots from home Takanori proclaimed: “Everything behaves as expected!” When asked about the phone call later he remembers: “It was 4:00 am on Tuesday morning May 25th. It was the time when I thought nothing happened. I was soundly asleep and least prepared for it to happen.” John recalls: “It was very busy. I spent a lot of time on the phone with Frank, talking to the run control shifter and the shift leader. It was a very exciting shift, but it wasn't until I was at home relaxing that the significance of what we'd done sunk in.”

All's well that ends well. Let's hope for many more high luminosity runs to come for all of us.

Rate plot as a function of time of the lowest energetic L2 trigger for electrons and photons (L1_EM2) before (red) and after veto (green). The difference between the two lines represents the dead time incurred by the large rate arising from just this one trigger. Once the HLT rejection was activated at 4:05 by disabling the L1 only selection (L1_ItemsStreamer, blue) the system stabilized quickly. Shortly after 6:15 the rate dropped below 200 Hz and the HLT rejection was disabled again.

Birgit Ewert

ATLAS e-News