Python CAN Bus API - Automate CAN/LIN Processing [Open Source]
Need to automate your CAN bus data processing via Python?
Vehicles and machines generate tons of CAN bus data - and GUI software is often not enough to extract the insights you need.
Here, Python scripts can help accelerate your data processing.
Here, you'll learn how to process CAN/LIN data via Python. In particular, we outline how to record/decode CAN data, key features of our Python API, use cases and case studies.
Tip: Try our Python API with J1939 data & DBC samples here.
How to record & decode CAN bus data?
To record CAN data, you can use a CAN bus data logger. For example, the CANedge1 lets you record raw CAN data to an SD card. Further, with the CANedge2, your data can be auto-pushed via WiFi to your own server.
The devices record CAN frames, which need to be decoded to physical values for analysis - typically via a DBC file.
Once the CAN bus data is DBC decoded, you can perform e.g. graphical visualization and statistical analysis.
In many cases, you may simply want to open a specific log file and analyze the data via a software GUI tool. For example, you might know that an issue occurred in a truck and you wish to review data related to the event manually.
For such use cases, various GUI software tools can be used. For the CANedge, you can either use the free asammdf GUI or load data into your favorite software tools:
Why use Python for CAN bus data processing?
GUI tools are great for small scale and ad hoc data analysis. However, cases exist where GUIs become impractical:
Too much CAN bus data
If you "record everything" from e.g. a truck, you're looking at 1 GB+ per day. If you wish to analyze 3 months of data for 10 trucks, that's almost 1 TB - not suited for GUI processing.
Too much manual work
If you are performing the same steps many times to e.g. extract a specific report/result from a GUI tool, you'll typically benefit from automating the process via scripts.
If you need to transform/process your data to enable advanced analyses - or integrate your data with a custom system/database - scripting is typically required.
Need for speed
If you need to extract results from your CAN data in "near real-time", you cannot rely on manual processes or GUIs - in particular if your use case needs to scale
You can use various scripting languages to process your CAN bus data. Python is popular as it enables efficient processing of data at scale, while also providing many plug & play packages suited for CAN data processing (e.g. our Python API modules).
Other popular options within the automotive sector include MATLAB. In particular, MATLAB's Vehicle Network Toolbox plugin lets you load MF4 log files from the CANedge natively for quick analysis, making it a powerful alternative if your team prefers to work within MATLAB. Further, MATLAB natively has great support for Parquet data lakes, meaning it's great for working with data that has already been DBC decoded via our MF4 decoders. On the other hand, Python has the benefit of being 100% free.
If you're new to Python, we recommend that you go through a few 'get started' tutorials. While not strictly necessary, it's a good investment to ensure you have the basics in place - and it'll definitely save you time in the medium/long run.
You can find several tutorials online, but we in particular recommend the Python tutorial videos by Mosh:
Once you've gone through the basics, you can try out some of our Python CAN API examples.
CAN bus Python API - key features
We offer an API with powerful features for processing your CAN/LIN data:
List log files between a start/stop time from local disk or your S3 server
Load MF4 files with CAN/LIN data into dataframes / iterators for processing
Load DBC file(s) and DBC decode raw CAN/LIN data to physical values
Modules are 100% free and open source FOSS - and supported on both Windows, Linux
Multiple simple-to-use code snippets are available to get you started quickly
Modules are efficient, yet extremely light with minimal dependencies
Use the CANedge Python API directly ...
The Python API gets you started quickly by handling "all the basics" involved in working with CAN bus data log files - including listing, loading and DBC decoding the data.
The Python CAN API is comprised of three separate Python modules - learn more below:
This module provides you with a list of CANedge log files based on a start/stop timestamp. The files can be listed from local disk or from an S3 server
This lets you load MF4 logs (or MFC/MFE/MFM) with CAN/LIN data into a pandas dataframe / iterator for processing
In practice, most users deploy a combination of these modules - and we generally refer to this combination as 'our Python API'. However, you can of course also use each of the modules in isolation for more specialized use cases.
% load, DBC decode and print MF4 log files
devices = ["LOG/958D2219"]
db_list = load_dbc_files(["dbc_files/j1939.dbc"])
fs = setup_fs(s3=False)
log_files = canedge_browser.get_log_files(fs, devices)
proc = ProcessData(fs, db_list)
for log_file in log_files:
df_raw = proc.get_raw_data(log_file)
df_phys = proc.extract_phys(df_raw)
... or start with a Parquet data lake
As an alternative, you can also use our MF4 decoders to DBC decode your log files into a Parquet data lake. By pre-processing the data, you can dramatically reduce the Python coding required in order to extract your insights. In particular, this approach is useful if your goal is to perform large-scale data analysis, where Python offers great Parquet support. Learn more about creating your own Parquet data lake in the MF4 decoder intro.
The above options are great for most CAN related use cases involving data from the CANedge. But if your goal is specifically to output modified MF4 files, then the asammdf Python API may be relevant.
The asammdf Python API lets you perform all of the steps you can do through the asammdf GUI, but through Python scripts. This means that you can load log files, DBC decode them and get the output into pandas dataframes - similar to our Python API tools.
For many use cases, you can thus use the asammdf API, though there are some distinct advantages to our API if you're working with data from the CANedge:
- Listing data: The asammdf API does not provide any methods for listing log files in a structured manner. In contrast, our Python API is designed to enable time-based listing of log files across both local disk and S3 (for use with the CANedge2)
- Support for MF4/MFC/MFE/MFM: The asammdf API lets you natively load CANedge log files in the MF4 format. However, if you wish to compress and/or encrypt your log files, you cannot natively load these log files via the asammdf API, but instead you'll need to first use the 'mdf2finalized' MF4 converter to get the files into the MF4 format
- Dependencies: The asammdf Python API enables a lot of features not included in our Python API. This also means that the tool is far heavier in terms of dependenices.
- AWS Lambda: If you wish to set up automated processing of incoming log files from the CANedge2, we provide ready-to-use AWS Lambda 'ARN layers' for enabling easy automation using our Python API
On the other hand, the asammdf API is better suited when your use case involves loading MF4 log files with CAN bus data and exporting the data into e.g. DBC decoded and/or concatenated MF4 log files. Similarly, the asammdf API also makes it easy to export data into MATLAB compatible formats. If in doubt which API to use, contact us.
Below we list a couple of other Python CAN tools that you may find useful. Note that most of the tools listed are intended for use with CAN bus interfaces (i.e. real-time streaming of data via USB), in contrast to our Python API and the asammdf API (intended for processing log files):
- canmatrix: This module lets you load CAN databases (e.g. DBC files) for use in scripts or for exporting to other formats. This module is used in our API and the asammdf API
- cantools: Like canmatrix, cantools enables loading of DBC files (and e.g. KCD, SYM, ARXML), encoding/decoding of CAN frames and basic graphical plots. It can e.g. be used with python-can to provide DBC decoding of raw CAN bus data streams
- python-can: Supports many CAN bus interfaces (incl. SocketCAN), enabling real-time streaming/transmission of raw CAN frames. Useful for diagnostics, testing etc. The module also supports various popular CAN bus file formats like Vector ASC, BLF, TRC
For more Python CAN tools, see this overview.
Use case examples: CAN bus + Python API
Visualize your CAN data via telematics dashboards
You can use Python within e.g. AWS Lambda functions to perform DBC decoding via the CANedge Python API or using our MF4 decoders - enabling easy integration with e.g. Grafana dashboards.
Handle ISO TP multiframe across J1939, NMEA 2000 and UDS
Our API examples provide basic Python classes for 'transport protocol' handling across popular CAN protocols - incl. combining frame sequences and decoding the data to physical values.
Integrate your CAN bus data into any system/database
Using the Python API, you can easily extract relevant CAN signals into pandas dataframes - making it easy to integrate with BI systems, databases etc. via standard Python APIs.
Perform advanced statistical analysis & diagnostics
Once you have your DBC decoded data available in a pandas dataframe, you can utilize a vast range of open source Python packages to help perform advanced statistical analysis.
Handle huge amounts of CAN data - e.g. for ML
Need to handle gigabytes or terabytes of CAN data - e.g. for machine learning? Python is ideal for this - in particular if you pre-process your data into a Parquet data lake.
Set up predictive maintenance to reduce breakdowns
Reduce breakdowns via simple predictive models (e.g. based on threshold alerts) and use event triggers (e.g. AWS Lambda functions) to auto-trigger the processing.
Reduce time spent on data processing through automation
Automate repetitive steps in your CAN bus analyses - for example creating PDF reports, extracting statistical results, performing CAN diagnostics or identifying events of interest.
Want to try working with our Python API in practice?
Check out our Python script examples - with free sample data and DBC files!
Case studies: Python API + CAN bus
Many of our end users deploy Python scripts for their CAN processing - see e.g. below examples from our 50+ case studies
Case study: CAN telematics
Learn how HAVELSAN use the CANedge2 to collect CAN bus data from their UGVs via SD/WiFi - and visualize it via dashboards.
"The CANedge2 is an autonomous way to collect, transfer & analyse data. Set it up - and watch the data from your office."full case study
Case study: Predict issues
Learn how one of our larger OEM customers use the CANedge2 to collect CAN bus data from heavy machinery in the field - and use it to perform Python based predictive maintenance.
"Both the hardware and support from CSS Electronics have been outstanding!"full case study
Case study: Dash dashboards
Learn how Nuvera Fuel Cells uses the CANedge2 and Python to perform automated diagnostics and reporting - with custom dashboard visualization.
"The CANedge2 is a very affordable and reliable solution for acquiring, elaborating and visualizing CAN bus data"full case study
Case study: ISO TP dashboards
Kverneland uses our Python API to process 2 x J1939 from the CANedge2, uploaded to their Azure cloud. In particular, they use our 'transport protocol' examples to handle J1939 ISO TP messages.
"The CANedge2 is a high quality product, but the support from CSS makes it an absolutely unbeatable solution!"full case study
Ready to process your CAN data via Python?
Get your CANedge today!