Python CAN Bus API - Automate CAN/LIN Processing [Open Source]
Need to automate your CAN bus data processing via Python?
Vehicles and machines generate tons of CAN bus data - and GUI software is often not enough to extract the insights you need.
Here, Python scripts can help accelerate your data processing.
Here, you'll learn how to process CAN/LIN data via Python. In particular, we outline how to record/decode CAN data, key features of our Python API, use cases and case studies.
Tip: Try our Python API with J1939 data & DBC samples here.
How to record & decode CAN bus data?
The Controller Area Network (CAN bus) is used in most automotives today (cars, trucks, tractors etc), as well as maritime vessels and machines.
To record CAN data, you can use a CAN bus data logger. For example, the CANedge1 lets you record raw CAN data to an SD card. Further, with the CANedge2, your data can be auto-pushed via WiFi to your own server.
The devices record CAN frames, which need to be decoded to physical values for analysis - typically via a DBC file.
Once the CAN bus data is DBC decoded, you can perform e.g. graphical visualization and statistical analysis.
In many cases, you may simply want to open a specific log file and analyze the data via a software GUI tool. For example, you might know that an issue occurred in a truck and you wish to review data related to the event manually.
For such use cases, various GUI software tools can be used. For the CANedge, you can either use the free asammdf GUI or load data into your favorite software tools:
asammdf GUI
This is a general-purpose GUI software that lets you load MF4 log files, DBC decode them and plot the physical values.
asammdf intro
MF4 converters
If you prefer using software like CANalyzer, PCAN-Explorer etc. you can convert your MF4 files to the relevant format.
mf4 converters intro
Why use Python for CAN bus data processing?
GUI tools are great for small scale and ad hoc data analysis. However, cases exist where GUIs become impractical:
Too much CAN bus data
If you "record everything" from e.g. a truck, you're looking at 1 GB+ per day. If you wish to analyze 3 months of data for 10 trucks, that's almost 1 TB - not suited for GUI processing.
Too much manual work
If you are performing the same steps many times to e.g. extract a specific report/result from a GUI tool, you'll typically benefit from automating the process via scripts.
Integration required
If you need to transform/process your data to enable advanced analyses - or integrate your data with a custom system/database - scripting is typically required.
Need for speed
If you need to extract results from your CAN data in "near real-time", you cannot rely on manual processes or GUIs - in particular if your use case needs to scale
You can use various scripting languages to process your CAN bus data. Python is popular as it enables efficient processing of data at scale, while also providing many plug & play packages suited for CAN data processing (e.g. our Python API modules).
Other popular options within the automotive sector include MATLAB. In particular, MATLAB's Vehicle Network Toolbox plugin lets you load MF4 log files from the CANedge natively for quick analysis, making it a powerful alternative if your team prefers to work within MATLAB. Further, MATLAB natively has great support for Parquet data lakes, meaning it's great for working with data that has already been DBC decoded via our MF4 decoders. On the other hand, Python has the benefit of being 100% free.
If you're new to Python, we recommend that you go through a few 'get started' tutorials. While not strictly necessary, it's a good investment to ensure you have the basics in place - and it'll definitely save you time in the medium/long run.
You can find several tutorials online, but we in particular recommend the Python tutorial videos by Mosh:
Once you've gone through the basics, you can try out some of our Python CAN API examples.
CAN bus Python API - key features
We offer multiple options for processing your CAN/LIN data in Python:
LOAD RAW DATA
Load log files directly into python-can for flexible processing ...
DBC DECODE
Use DBC file(s) to efficiently decode raw CAN/LIN data to physical values
100% FREE
Modules are 100% free and supported on both Windows, Linux
API EXAMPLES
Multiple simple-to-use code snippets are available to get you started quickly
LIGHT WEIGHT
Modules are efficient, yet extremely light with minimal dependencies
Load your log files directly ...
The CANedge log file format can be loaded directly in the popular python-can library. This lets you directly analyse raw CAN/LIN frames, replay log files onto USB-to-CAN interfaces or e.g. selectively DBC decode data into physical values. It is extremely flexible and ideal for specialized use cases.
... or start with a Parquet data lake
As an alternative, you can use our MF4 decoders to DBC decode your log files into a Parquet data lake. By pre-processing the data you dramatically reduce the Python coding required in order to extract your insights. In particular, this approach is useful if your goal is to perform large-scale data analysis, where Python offers great Parquet support. Learn more about creating your own Parquet data lake in the MF4 decoder intro.
The above methods are currently the recommended and maintained methods for analysing your CANedge data in Python. In particular, our integration with python-can now replaces our previous Python API, which consisted of below sub modules. You can still continue using this legacy API, but it will not be further developed:
Our legacy Python CAN API is comprised of three separate Python modules - learn more below:
canedge_browserThis module provides you with a list of CANedge log files based on a start/stop timestamp. The files can be listed from local disk or from an S3 server
mdf_iterThis lets you load MF4 logs (or MFC/MFE/MFM) with CAN/LIN data into a pandas dataframe / iterator for processing
can_decoderThis module lets you decode raw CAN/LIN data using DBC files (or manually entered decoding rules) - incl. support for J1939, NMEA 2000, OBD2
In practice, most users deploy a combination of the above modules - and we generally refer to this combination as 'our Python API'. You can of course also use each of the modules in isolation for more specialized use cases. Today, however, we recommend to use our python-can integration or the MF4 decoders.
The above options are great for most CAN related use cases involving data from the CANedge. But if your goal is specifically to output modified MF4 files, then the asammdf Python API may be relevant.
The asammdf Python API lets you perform all of the steps you can do through the asammdf GUI, but through Python scripts. This means that you can load log files, DBC decode them and get the output into pandas dataframes - similar to our Python API tools.
For many use cases, you can thus use the asammdf API, though there are some distinct advantages to our API if you're working with data from the CANedge:
- Listing data: The asammdf API does not provide any methods for listing log files in a structured manner. In contrast, our Python API is designed to enable time-based listing of log files across both local disk and S3 (for use with the CANedge2)
- Support for MF4/MFC/MFE/MFM: The asammdf API lets you natively load CANedge log files in the MF4 format. However, if you wish to compress and/or encrypt your log files, you cannot natively load these log files via the asammdf API, but instead you'll need to first use the 'mdf2finalized' MF4 converter to get the files into the MF4 format
- Dependencies: The asammdf Python API enables a lot of features not included in our Python API. This also means that the tool is far heavier in terms of dependenices.
- AWS Lambda: If you wish to set up automated processing of incoming log files from the CANedge2, we provide ready-to-use AWS Lambda 'ARN layers' for enabling easy automation using our Python API
On the other hand, the asammdf API is better suited when your use case involves loading MF4 log files with CAN bus data and exporting the data into e.g. DBC decoded and/or concatenated MF4 log files. Similarly, the asammdf API also makes it easy to export data into MATLAB compatible formats. If in doubt which API to use, contact us.
Below we list a couple of other Python CAN tools that you may find useful. Note that most of the tools listed are intended for use with CAN bus interfaces (i.e. real-time streaming of data via USB), in contrast to our Python API and the asammdf API (intended for processing log files):
- canmatrix: This module lets you load CAN databases (e.g. DBC files) for use in scripts or for exporting to other formats. This module is used in our API and the asammdf API
- cantools: Like canmatrix, cantools enables loading of DBC files (and e.g. KCD, SYM, ARXML), encoding/decoding of CAN frames and basic graphical plots. It can e.g. be used with python-can to provide DBC decoding of raw CAN bus data streams
- python-can: Supports many CAN bus interfaces (incl. SocketCAN), enabling real-time streaming/transmission of raw CAN frames. Useful for diagnostics, testing etc. The module also supports various popular CAN bus file formats like Vector ASC, BLF, TRC
For more Python CAN tools, see this overview.
Use case examples: CAN bus + Python API
Visualize your CAN data via telematics dashboards
You can use Python within e.g. AWS Lambda functions to perform DBC decoding via the CANedge Python API or using our MF4 decoders - enabling easy integration with e.g. Grafana dashboards.
Handle ISO TP multiframe across J1939, NMEA 2000 and UDS
Our API examples provide basic Python classes for 'transport protocol' handling across popular CAN protocols - incl. combining frame sequences and decoding the data to physical values.
Integrate your CAN bus data into any system/database
Using the Python API, you can easily extract relevant CAN signals into pandas dataframes - making it easy to integrate with BI systems, databases etc. via standard Python APIs.
Perform advanced statistical analysis & diagnostics
Once you have your DBC decoded data available in a pandas dataframe, you can utilize a vast range of open source Python packages to help perform advanced statistical analysis.
Handle huge amounts of CAN data - e.g. for ML
Need to handle gigabytes or terabytes of CAN data - e.g. for machine learning? Python is ideal for this - in particular if you pre-process your data into a Parquet data lake.
Set up predictive maintenance to reduce breakdowns
Reduce breakdowns via simple predictive models (e.g. based on threshold alerts) and use event triggers (e.g. AWS Lambda functions) to auto-trigger the processing.
Reduce time spent on data processing through automation
Automate repetitive steps in your CAN bus analyses - for example creating PDF reports, extracting statistical results, performing CAN diagnostics or identifying events of interest.
Export data to custom formats for use in 3rd party tools
Need to convert CAN log files into other formats? If your format is not supported by our MF4 converters or decoders, Python can be used to load log files and export them to custom formats
Want to try working with our Python API in practice?
Check out our Python script examples - with free sample data and DBC files!
Case studies: Python API + CAN bus
Many of our end users deploy Python scripts for their CAN processing - see e.g. below examples from our 50+ case studies
Case study: CAN telematics
Learn how HAVELSAN use the CANedge2 to collect CAN bus data from their UGVs via SD/WiFi - and visualize it via dashboards.
"The CANedge2 is an autonomous way to collect, transfer & analyse data. Set it up - and watch the data from your office."
full case study
Case study: Predict issues
Learn how one of our larger OEM customers use the CANedge2 to collect CAN bus data from heavy machinery in the field - and use it to perform Python based predictive maintenance.
"Both the hardware and support from CSS Electronics have been outstanding!"
full case studyCase study: Dash dashboards
Learn how Nuvera Fuel Cells uses the CANedge2 and Python to perform automated diagnostics and reporting - with custom dashboard visualization.
"The CANedge2 is a very affordable and reliable solution for acquiring, elaborating and visualizing CAN bus data"
full case study
Case study: ISO TP dashboards
Kverneland uses our Python API to process 2 x J1939 from the CANedge2, uploaded to their Azure cloud. In particular, they use our 'transport protocol' examples to handle J1939 ISO TP messages.
"The CANedge2 is a high quality product, but the support from CSS makes it an absolutely unbeatable solution!"
full case studyReady to process your CAN data via Python?
Get your CANedge today!
