CAN Bus Troubleshooting at CERN [CANopen WiFi Gateway]

What problem did you solve?

At the CERN LHC experiment, several gaseous detectors are present - each one having a 'Gas Control System' (around 30 in total) responsible for providing the correct gas mixture to the detectors. We found out that one such systems started showing spikes in the flow of the single channels sending the gas to the detectors.

After confirming that there was no physical explanation for the spikes we had to identify the root cause. The readout of flowmeters (called 'flowcells' in our case) is done by making the gas pass through a channel where a PT100 is installed. By leveraging the thermal conductivity that changes with the flow of the gas we are able to compute the real flow passing through each channel.

The data is read using a CERN Embedded Local Monitoring Board (ELMB) which is a multi function board providing digital/analog inputs and outputs. The board has a CAN bus interface. We wanted to understand if these spikes originated from an ELMB, e.g. due to a board malfunction or firmware issue.

To achieve this, we needed a CAN logger to sniff all the messages passing through the CAN bus, in order to compare this data vs. the archived data in our databases.

How did you solve it?

We first setup an object storage service on CERN's openstack infrastructure. In particular, we created an S3 storage bucket. Next, we configured a CANedge2 and let it access CERN's WiFi network so that it could directly upload data to the S3 bucket.

Finally, we inspected the uploaded CANopen data using the CANedge Python API, leveraging the available API examples to get started quickly.