MATLAB - Analyze Your CAN Bus Data [Parquet, MDF]
Need to analyze your CAN bus data in MATLAB?
In this overview, you'll learn how to load your CAN bus data into MATLAB - either as MF4 files (MDF) or DBC decoded Parquet files. We also outline key features of MATLAB's Vehicle Network Toolbox and Predictive Maintenance Toolbox.
How to log & interpret CAN bus data?
The CAN protocol is today used in practically all automotive vehicles (cars, trucks, buses, forklifts, ...) and in many industrial machinery applications. This makes CAN data important to many use cases, such as vehicle diagnostics, fleet management, predictive maintenance, academic research and more.
To record CAN bus data (incl. higher layer protocols), you can use a CAN data logger. For example, the CANedge1 lets you log CAN data to an SD card, while the CANedge2/CANedge3 also enable upload of the data to your own server via WiFi/LTE.
The devices record 'raw' CAN frames, which need to be decoded to physical values for analysis - typically via a DBC file. Once the CAN bus data is DBC decoded, you can perform e.g. graphical visualization and statistical analysis.
In some cases you may simply want to open a specific log file and analyze the data via a software GUI tool. For example, you may need to analyze a specific event or issue in a vehicle.
For such use cases, various GUI software tools can be used. For the CANedge, you can for example use the free asammdf GUI - or convert & load data in your favorite GUI tools:
Why use MATLAB for CAN bus data processing?
MATLAB is popular tool for analyzing CAN bus data amongst automotive OEMs (Original Equipment Manufacturers). In particular, MATLAB is popular for large-scale automotive fleet testing, algorithm development and predictive modelling.
Looking at a typical data analytics workflow, MATLAB offers several benefits:
Load and explore CAN bus data
Load CAN data from various log file formats (Parquet, MF4, MAT, ...) and locations (local, S3, ...). Data can be loaded as e.g. single file, datastores or tall arrays - for exploration via tables, stats & plots
DBC decode and preprocess data
DBC decode raw CAN data to physical values via the Vehicle Network Toolbox. Further, easily clean and pre-process your decoded CAN bus data via smoothing, filling, outlier handling, rescaling etc.
Develop models & analyses
Datastores and tall tables let you easily switch between analyzing a single device, trip or signal - to performing statistical analysis across fleets or months of data, incl. via e.g. machine learning
Deploy & integrate in your system
MATLAB offers the expertise and tools for deploying your CAN bus analytics, from development to production - for example to implement predictive maintenance models into your fleet operations
Why use the CANedge with MATLAB?
The CANedge is ideal for OEM engineers using MATLAB:
The hardware is low cost, meaning several devices can be deployed for e.g. prototype vehicle fleet tests. At the same time, it's extremely easy to work with your CANedge data in MATLAB - either by directly loading the MF4 log files, or by connecting MATLAB to your Parquet data lake of DBC decoded data.learn about the CANedge
How to load your CAN bus data into MATLAB
Directly load unfinalized MF4 logs from the CANedge
With MATLAB 2021b and the Vehicle Network Toolbox, you can directly load MF4 log files from the CANedge with raw CAN/LIN data. The MF4 file can then be finalized in MATLAB via a simple function call. This approach makes it easy to load specific log files in MATLAB for e.g. vehicle diagnostics and automotive testing purposes.
Load 'finalized' MF4 files - incl. via datastores
You can also use the Vehicle Network Toolbox to load 'finalized & sorted' MF4 files. To do this, you can use our MF4 converter to perform the finalization & sorting step prior to loading your data (a process which can be easily automated). You can also load finalized MF4 files into MATLAB datastores to easily work across large numbers of log files.
Load DBC decoded MF4 files - incl. via tall arrays
The VNT also lets you load MF4 log files that have been DBC decoded to physical values outside MATLAB. The DBC decoding can be done via the asammdf GUI or automated API scripts as per our examples. In this case, you can also use tall arrays, ideal for "big data" use cases - like fleet testing across 100+ devices where the terabytes of data do not fit into memory.
Load DBC decoded Parquet data lake [recommended]
You can use our MF4 decoders to easily create a Parquet data lake of DBC decoded CAN/LIN data - which can natively be processed within MATLAB (no toolbox required). MATLAB also supports Parquet tall arrays (stored locally or on S3) for out-of-memory operations - as well as powerful time resampling functions. This is our recommended integration method.
Easily load data from S3
If you're using the CANedge2/CANedge3, you may wish to load log files directly from your S3 server. This can be done in multiple ways: Within MATLAB, you can create a datastore from an S3 bucket based on DBC decoded MF4 or Parquet files. Alternatively, you can mount your S3 bucket as a local drive using e.g. TntDrive.
MATLAB's Vehicle Network Toolbox
MATLAB's Vehicle Network Toolbox offers powerful features for working with your CAN data:
Easily load MF4 log files from the CANedge
Load folders / buckets and work with 'datalakes'
Easily analyze huge datasets (beyond memory)
Load files / datastores directly from S3 servers
Decode raw CAN bus data using DBC files incl. J1939
Easily integrate with your existing systems
The Vehicle Network Toolbox makes it easy to DBC decode raw CAN bus data to physical values. You can either extract tables containing all decoded signals - or extract only specific signals for further analysis.
MATLAB does not currently support transport protocol decoding e.g. for J1939 multiframe messages, NMEA 2000 fast packets or UDS (Unified Diagnostic Services). If this form of decoding is required, we recommend that you use our Python API (with our ISO TP Python class) or use the asammdf Python API and then export DBC decoded data to e.g. MF4, CSV or MAT.
Example: MF4 CAN bus tall arrays
One of the powerful features of the Vehicle Network Toolbox is ability to load MF4 log files into MDF tall arrays. Effectively, this lets you analyze an unlimited number of log files with a few lines of code - regardless of the total file size.
Further, tall arrays can be plotted directly via MATLAB's visualization tools like plot, histogram and histogram2 - enabling quick visualization of e.g. months of data.docs script example
Case study: Truck telematics
"Excellent functionality. Outstanding support. Continuous improvements. The CANedge2 is the best you can get!"full case study
Predictive maintenance with the CANedge and MATLAB
CANedge2 for predictive maintenance
The CANedge2 is an ideal device for predictive maintenance as it lets you easily record CAN/LIN data - and automatically push this to your own S3 server for processing.
The raw CAN data is recorded to an SD card, meaning that a periodic loss of connectivity does not result in loss of data (e.g. if vehicles operate in mines, tunnels, outside of WiFi range etc). Further, data can be uploaded via stationary WiFi access points using the CANedge2 or via 3G/4G using the CANedge3 (for on-the-road upload of data). Log files can be split in configurable chunks based on size or time - e.g. in 1 MB splits, 10 second splits, 30 min splits etc.
We encounter two common ways to utilize the CANedge in predictive maintenance:
#1 Predictive model training
When training predictive models (e.g. via machine learning), it is vital that the data sets are "rich" and "complete". Here, the CANedge2 is ideally suited as you can record "everything" from 2 x CAN/LIN - sometimes comprising 4000+ frames/second. This amount of data can correspond to 2 GB+ per day - making it too costly for 3G/4G transfer, but often fine for periodic offload via WiFi routers in e.g. a garage. Via the device configuration, you can of course optimize your logging via filters, prescalers, triggers and compression.
In this type of use case, the CANedge2 may be deployed in e.g. 100-300 vehicles over a period of 6-24 months. By collecting operational data (such as speed, RPM, temperatures) along with "failures" (e.g. diagnostic trouble codes), you can collect many TB of 'failure data' directly in S3 for e.g. machine learning models
#2 Operational deployment
The CANedge2/CANedge3 can also be used in production deployments of predictive maintenance models. Here, the upload frequency is typically increased (e.g. to 1 minute or 5 minute intervals) and the incoming data is often automatically decoded, analyzed and reacted upon through event triggered scripts (using e.g. AWS Lambda functions).
Use cases often start with basic predictive models based on threshold analyses: "Does parameter X go above limit Y for Z seconds, while parameter A is below limit B?".
This type of "practical" predictive maintenance is typically a good way to start and many of our users successfully predict e.g. battery fires in forklifts, tire issues etc. via such methods.
MATLAB's Predictive Maintenance Toolbox
As use cases scale and the sophistication requirements increase, it becomes increasingly important to train and deploy more advanced predictive models. Here, tools like the MATLAB Predictive Maintenance Toolbox can be extremely useful:
Estimate an asset's time to failure via Remaining Useful Life (RUL) predictions
Use survival, similarity, trend based or ML models to estimate the RUL
Easily integrate CAN data MF4 log files via the Vehicle Network Toolbox
Take outset in existing reference models for e.g. gearboxes, batteries, motors
Detect system anomalies and faults via e.g. control charts or Kalman filters
Deploy predictive models to edge devices or cloud e.g. via generated C/C++ code
As indicated above, predictive maintenance for CAN bus data can sound simple enough: Evaluate signal values vs. defined threshold limits to determine if an issue is about to occur. If an issue occurs, trigger an alert.
While simple models can be set up this way, practical deployment across a fleet of vehicles or machines will typically require more advanced modelling. Even basic things like proper data preprocessing can be make-or-break to avoid countless "false alert" situations.
Here, using the CANedge2 while partnering with MATLAB provides an excellent combination of low cost hardware and access to vital support and useful tools.
We provide basic support in regards to loading your CAN bus data into MATLAB (see our script examples). Further, we can typically answer basic questions and we're always happy to provide high level sparring on your use cases.
For more technical support on your MATLAB script, we recommend to reach out to the MATLAB support team directly, as their experts will be better suited to diagnose any issues and help you quickly progress.
You can load MF4 log files from the CANedge and replay them within the MATLAB/Simulink environment, see this article for details.
As per our Python API article, we also offer a Python API for use with the CANedge CAN bus data loggers. The API lets you do many of the same things as MATLAB and Python and our API are 100% open source and free.
However, if you're going to perform advanced analyses or if you're already familiar with MATLAB, you'll probably be better off going with MATLAB. In particular, this route has the major benefit of enabling support and expertise from the MATLAB team. When it comes to e.g. data preprocessing and predictive modelling, it's not only a matter of 'whether' something can be done in e.g. Python or MATLAB - but rather 'how' it can be done. Implementing advanced models in the best way can be hard to do on your own - and here MATLAB's team of experts can help provide outside-in sparring based on similar experiences across other automotive end users.
We also have users that start with simple use cases in Python and later on move to MATLAB. At the end of the day, you can decide on your own as the CANedge and related MF4 data is designed to be interoperable - letting you manage your data in the way you prefer.
Ready to process your CAN data in MATLAB?
Get your CANedge today!