MF4 (MDF4) Explained - A Simple Intro [+J1939/OBD2 Data]

MDF Measurement Data Format Intro CAN Bus Data Logger

What is an MF4 file (aka MDF4) - and why is it key to CAN bus data logging?

The ASAM MDF (Measurement Data Format) is a binary file format for recording e.g. CAN, CAN FD and LIN bus data. Today, the MDF format is the industry standard - ensuring interoperability across many CAN tools.

The CANedge supports the latest MF4 file format - and below we list benefits, basics and software/API tools. We also provide sample MF4 data (J1939, OBD2).



What is ASAM MDF?

MDF was developed in the 90's by Vector GmbH and Robert Bosch GmbH. Initially it was proprietary, but in 2002 it was made public - and in 2009 it was standardized by ASAM. The latest version, MDF 4.11, was released in 2014.

Today, the MDF file format is the de facto industry standard - and it is used in an extensive suite of CAN software, APIs - and hardware like the CANedge1, CANedge2 and CANedge3.

CANedge MDF4 CAN Bus Data Logger

6 key benefits of MF4

Logging data as MDF4 files has several major benefits:

ASAM MDF open standard software support INTEROPERABLE

MDF is open, standardized & widely supported for CAN bus logging - ensuring interoperability across many tools, software and APIs

MDF4 fast writing CAN logger frames second lossless FAST WRITING

Logging CAN data is a challenge due to increasing data rates. MDF is designed for fast writing - enabling lossless recording of e.g. CAN FD

MDF efficient binary file format reader FAST READING

The binary MDF is index based and extremely fast to read, edit or e.g. DBC convert - making it ideal for databases and big data analytics

MDF deflate compress zip compact file optimized COMPRESSIBLE

MDF 4.1 supports compression, reducing file size drastically. This improves SD capacity, server costs & 3G/4G upload costs

Measurement Data Format Raw vs DBC Conversion Physical Values RAW/PHYSICAL

MDF4 supports both "raw" bus logging and "physical" values - hence you can often use the same software/ APIs

MDF MF4 free software tool API asammdf github SOFTWARE/API

The format is supported by free powerful software & APIs. It's e.g. easy to DBC convert MDF4 data and export it to CSV, MATLAB, pandas

These features make MDF ideal for e.g. cloud analytics and telematics - across CAN, CAN FD, LIN, J1939, OBD2, CANopen, NMEA 2000 and more.



MF4 file structure

In this section, we provide a quick overview of the basic MDF4 file structure - with outset in a CANedge2 raw CAN bus log file.

MDF4 files are comprised of "blocks" - most of which are in practice arbitrarily ordered. Via a system of internal "links", these blocks form a hierarchy.

This structure enables more flexible log files vs. e.g. CSV - ideal for e.g. mixing CAN, CAN FD & LIN or supporting both raw/physical data.

For technical details on MDF files, expand the below:

MDF File Structure Overview Illustration



Measurement Data File Example Vector MDF Validator

To visualize the MDF4 hierarchy in practice, we can take outset in a raw CAN bus log file. In the screenshot, we've used Vector's free MDF Validator tool to load a raw J1939 MDF4 log file (*.MF4) from the CANedge2.

This is in general a useful tool for understanding the basic structure of MDF files - and checking their validity.




As evident from the above, there are several key MDF blocks:

Block Description
Identification (ID) 64 bytes block at the beginning of an MDF. Identifies file as MDF and specifies version
Header (HD) Adds info after ID (e.g. log start time). Acts as file root and links to each Data
File History (FH) Describes tools involved in creating the file (incl. e.g. the CANedge FW version)
Data Group (DG) Contains measurement data incl. Data Blocks and Channel Groups
Data (DT) Contains records (e.g. CAN frames). The CANedge stores all raw data in one DT
Channel Group (CG) Describes the layout of records. The CANedge uses 8 CGs (2 per CAN/LIN channel)
Channel (CN) A Channel Group contains channels that describe the data structure and signals



Note: MDF "channels" should not be confused with "bus channels" (the CANedge e.g. has 2 x CAN and 2 x LIN "bus channels").

If you're logging raw CAN bus data via "bus channel" 1 on the CANedge, the data will be stored in Channel Group 1 and 2. Here, CG 2 enables the use of variable data length (necessary for e.g. CAN FD).

Within CG 1, you'll see the CN CAN_DataFrame. This is split in 7 parts/sub channels:

  1. BusChannel: The bus channel that the data is from
  2. ID: The CAN ID
  3. IDE: The ID type (regular, extended)
  4. DLC: The CAN frame DLC
  5. DataLength: The actual data length (may differ from the DLC for CAN FD)
  6. DataBytes: The data bytes of the CAN frame
  7. Dir: The direction (received, transmitted)
MDF CAN bus logging data example

The CANedge writes log files as MDF 4.11 using the 'bus logging' concept to create log files with raw CAN/LIN data.

To ensure fast writing, the CANedge appends every new CAN/LIN frame to a single Data Block. This results in an "unsorted" MDF file. To enable fast reading in e.g. your API processing you may choose to sort the MDF file before/during processing. Doing so in advance is not required (as most tools support unsorted MDFs) - but it enables fast indexing and hence faster processing.


If you sort a raw CANedge log file using e.g. the asammdf GUI/API, you'll note that the resulting MDF file is structured differently from before - see the MDF Validator example.

Instead of one Data Group, there is now a Data Group for each bus channel.

Further, there are no longer two Channel Groups per bus channel. Instead, there's a single Channel Group with two Data blocks (contained in a Data List).

This structure ensures that each Data Group contains Data blocks with records of identical layout - enabling fast index based reading.

Sorted MDF CAN Bus Channel 1

In some cases you may want to trigger a small "sorting" script everytime a new raw log file is uploaded to your S3 server. This way you can store only sorted files on your server - enabling faster reading/processing of the files when needed.

You can also take this a step further and automatically DBC convert your raw MDF log files as they come into your server. This process will include the sorting step and enable you to directly fetch human-readable data for e.g. plots. For examples of how to sort and/or DBC convert your MDF data, see our API sample library and the examples below.


The below links can be useful in order to gain a further understanding of MDF:





Raw vs. physical (human readable) MDF4 data

When you connect the CANedge to your car, truck or other CAN/LIN application, you'll record raw bus data.

To work with the data, you'll need to convert it to human-readable data - aka "physical" or "scaled" values.

In order to convert your raw MDF files, you'll need a database of conversion rules - e.g. a DBC file (CAN database).

With a DBC, you can use e.g. asammdf (see below) to convert your raw data. The resulting MDF file contains a Data Group for each CAN ID and a Channel for each message signal.

You can then easily plot the decoded data in e.g. asammdf - or export it to csv/mat for analysis in Excel/MATLAB.

DBC convert MDF bus logging data CAN bus J1939 OBD2




MDF4 software & API tools

So how do you actually manage your CAN bus MDF4 log files?

Below we list examples of software/APIs that you can use to manage your MDF4 data.

TIP: If you're new to MDF, we recommend the open source asammdf GUI/API and the simple MDF4 converters to get started.

Free MDF4 Software Tool API


asammdf (GUI)

The open source asammdf offers a simple-to-use GUI for ad hoc reading, editing, DBC conversion and plotting of MDF files

  • Raw/converted MDFs
  • Concatenate, resample
  • DBC convert (J1939, OBD2)
  • Graphical plots
  • Export (CSV, MATLAB, ...)
  • 100% open source
  • Windows/Linux support

Learn more

MF4 converters

Our free open source MF4 converters let you easily convert raw CANedge data to popular formats:

  • Output: ASC, TRC, CSV, ...
  • Finalize & sort MDF4 files
  • Drag & drop + CLI
  • Easily use in scripts
  • 100% open source
  • Windows/Linux support

Learn more

MF4 decoders

Our free open source MF4 decoders let you easily DBC decode raw CANedge data to popular formats:

  • Output: DBC decoded CSV, Parquet
  • Finalize & sort MDF4 files
  • Drag & drop + CLI
  • Easily use in scripts
  • 100% open source
  • Windows/Linux support

Learn more

CANedge MF4 Python API

We provide an open source MF4 Python API:

Python API intro

MDF Validator

This free MDF viewer from Vector lets you load raw/converted MDF files to view the structure and evaluate validity. Useful for understanding the MDF file.

  • 100% free (requires your details)
  • Windows support
  • View MDF structure & meta info
  • Check the validity of an MDF file
  • Load raw & converted MDF files

Learn more

Vector MDF Validator Measurement Data Format Files

Vector Tools (CANalyzer)

CANalyzer natively supports finalized & sorted MDF4 files.

  • License based
  • Comprehensive analysis tool
  • Advanced graphical plots
  • Load raw & converted MDF files

CANalyzer details

MDF4 CANalyzer Low Cost Hardware

HBM Tools (nCode)

The nCode software is popular within equipment analysis, e.g. for durability testing and big data visualization.

  • License based
  • Comprehensive analysis tool
  • Advanced graphical plots
  • Load raw & converted MDF files

nCode details

HBM nCode MDF CAN Bus Analysis Software Tool

asammdf (Python API)

The API behind the asammdf GUI tool provides a full suite of all the core operations you need to process both raw and converted MDF files at scale.

  • 100% free & open source
  • Load raw & converted MDF files
  • Concatenate, filter, resample, ...
  • DBC conversion (incl. J1939, OBD2)
  • Extremely fast processing
  • Graphical plots
  • Export (CSV, pandas, MAT, HDF5, ...)
  • Easily combine with S3 boto3 API

asammdf intro | API examples

MDF4 Python API Script Programming Automation

Vehicle Network Toolbox (MATLAB API)

MATLAB provides a powerful toolbox for analysis of MDF4 data with focus on automotive use cases

  • License based MDF import
  • Load finalized & converted MDF files
  • DBC conversion
  • Efficient handling of large files
  • Powerful plot functionality
  • Work directly with files on S3

VNT intro

MATLAB Vehicle Network Toolbox MDF4 MF4 ASAM

Other

There are many more MDF software tools available for use with the MDF file format. Examples include tools from dSpace (AutomationDesk, ControlDesk), ETAS (ASCMO, INCA) and many more.

ASAM MDF software directory

ASAM MDF Software Tools Directory




Example: J1939 MF4 data (tractor)

The CANedge is excellent for use with heavy duty vehicles.

In particular it's ideal for vehicle fleet telematics across e.g. trucks, transit buses, excavators, tractors.

The asammdf GUI/API supports decoding J1939 data via J1939 DBC files. To test this, download the sample data below (*.MF4 & DBC) and load it in the asammdf GUI.

J1939 telematics Sample data J1939 DBC
J1939 MDF Sample Data Free Download *.mf4




Example: OBD2 MF4 data (Audi A4)

Vehicle Data ASAM MDF Sample OBD2 DBC Practical

Another popular use case is recording OBD2 data from cars.

Most cars support OBD2 data, letting you analyze e.g. speed, fuel consumption, brake patterns and diagnostics.

A challenge with OBD2 can be in converting the raw data. However, this is easily done with asammdf and our OBD2 DBC. Try it out via the sample files below:

OBD2 logging Sample data OBD2 DBC



Ready to log your CAN/LIN data in the MF4 format?

Get your CANedge today!







Recommended for you