ChatGPT: Analyse 1Y of EV6 CAN Bus Data [Code Interpreter]

Using asammdf or our MF4 decoders is the simplest way to get started with your own data. In our case, however, we must use the Python API because our data contains multiframe UDS response data from the Kia EV6 (not currently supported by the asammdf GUI).

In theory it should be possible to allow GPT4 to perform the DBC decoding and data processing by providing it with the relevant Python packages in the form of *.whl files and some guidance. We did not test this particular step, however, as the purpose in this article is to focus on how competent GPT4 is when analysing a 'prepared' dataset.

Similarly, GPT4 can handle multiple CSV files rather than a single CSV file (as shown later). However, GPT4 achieves this by looping through the files and we expect it would increase the delay significantly between prompt responses, hindering the workflow. In addition, by performing the CSV concatenation prior to upload, we create a more consistent starting point for the analysis, while making it easier to double-check the results on our side.

When exporting DBC decoded CAN bus data to CSV, we typically distinguish between two types of data structures:

1: Signals in rows

Here, a 'Signal' column contains the name of each signal (e.g. Speed, RPM, ...). This lets you efficiently store signals with different timestamps and frequencies. This can be useful for CAN bus data where each CAN message generally has a unique timestamp/frequency. However, it also makes it cumbersome to e.g. calculate ratios between signals on-the-fly.

You can produce this CSV structure via our Python API.

2: Signals in columns

Here, each column represents a unique signal, all sharing the same timestamp (found in the index column). To represent signals across multiple CAN messages in this structure, one has to use resampling and/or extrapolation. For example, one could resample each signal to a shared 5-second frequency.

You can produce this structure via our Python API and the asammdf GUI.

Generally, the first method is superior from a technical perspective as you can minimize resampling/extrapolation, thus retaining the best data quality. For example, resampling a 100 Hz acceleration signal to a 5 second frequency results in this signal losing almost all meaning. Conversely, resampling an outdoor temperature to a 5 second frequency implies zero practical loss in quality. With the first data structure, you can address these considerations dynamically, tailoring your approach to each analysis. Further, for some use cases it is critical that data is analysed at the original frequency (e.g. in some automotive diagnostics use cases) and here this structure is recommended.

With that said, if you are going to analyse CAN data across e.g. 1 year, then you'll most likely need to do some level of resampling before exporting it to CSV (to avoid analysing e.g. 5-10 GB of data per device). For such a use case we generally recommend the second structure.

For our use case, we want to end up with a single CSV file (signals in columns) that has a consistent set of headers. This could be achieved by looping through all the log files and loading them into a single pandas dataframe, then saving this. Given that our data might exceed available RAM, this approach may not be scalable.

Instead, we append data to a CSV file that accumulates with each loop iteration. To do this, a list of signals needs to be defined up front which must be available in each dataframe. Without this, we cannot ensure that each value appended ends in the correct column of the CSV.

Basically, the signal list will be used to filter each dataframe (to ensure we do not get excess signals), as well as discard dataframes (if they do not include all required signals, e.g. because of a very short file, differences in configuration etc.).

In order to identify this list of signals, we take the following steps:

• Identify a log file that should be 'representative' of the majority of the data
• Extract and restructure the dataframe to have signals in columns
• List the column names using list(df_phys) and subsequently sort the list
• Use this list as the preliminary signals_required variable and filter all decoded logs
• Next, loop through all the data and extract the list of column names for each file
• If the column names of a file differs from the signals_required, print a warning and review the reason for the difference, then take action accordingly

Upon iterating, you might find that while 30% of the data aligns 95% with your 'signals_required', there are discrepancies, possibly due to 1-2 redundant signals. By storing the signals_extracted in a new list, you can quickly look for the unique elements between that and your signals_required via below line:

list(set(signals_required).symmetric_difference(set(signals_extracted)))

You then update your signals_required list to remove the redundant signals and repeat until you've arrived at a satisfactory signals_required list that also lets you work with e.g. 80-90% of your data. Of course, some use cases may require a different approach.

Before uploading the large CSV file to GPT4 it can be useful to do a quick review of the data.

For example, do some sanity checks by loading the file in Excel (if possible) - or create a smaller output file based on e.g. just 20 logs, then upload it to GPT4 for initial feedback.

For example, ChatGPT informed us that the signal VehicleSpeed had several missing values. This is a known problem with UDS responses from the Kia EV6 (and other Hyundai/Kia EVs) which is why we'd normally discard this parameter in favour of the Speed signal produced by the CANedge GPS module (which is consistently reliable). We therefore updated the output file to remove VehicleSpeed.

In the following analyses, we adhere to the following restrictions:

• We do not use any coding knowledge in any of our prompts
• Each analysis was completed in <30 min (most in <10 min)
• All outputs you'll see are directly from ChatGPT (with no edits by us)
• We started a new 'chat session' for most of the examples to get a clean slate
• The analyses were done in August 2023 (ChatGPT August 3 version)

Below is the full ChatGPT workflow (excl. the Python code generated), followed by the plot insights it provides.

As per the workflow, ChatGPT defines a list of tasks for each part of our prompt, then produces the code required to deliver the necessary output. The code it generates sometimes fails and ChatGPT then uses the error information to refine the code and try again. At one point we need to give it a 'nudge' to continue, but other than that it succeeds in finishing the task in the first attempt.

ChatGPT also provides a summary conclusion of each chart. The first iteration is generic and not useful. However, when we give ChatGPT this feedback it delivers far more relevant insights:

Below is the full ChatGPT workflow followed by the plot insights it provides. As follow-up to our original prompt, we make a number of styling corrections and tweaks (e.g. converting Speed to SpeedKMH), but this is quickly handled by ChatGPT.

As shown, most of the insights make sense and are relevant.

Interestingly, however, GPT4 claims in the second insight that the EV6 StateOfHealth is deteriorating. That's a bold claim and seems inconsistent with the visual plot. After some probing, GPT4 admits that this was an error.

As we will see throughout other examples, this is an example of ChatGPT producing valid plots/numbers - but often invalid conclusions. Specifically, our hypothesis in this case is that ChatGPT 'solves' the insight aspect of our prompt based on its training data, rather than on the actual numbers - and it must be 'forced' to use the actual results in order to avoid this. In turn, this also begs the question as to whether the conclusions ChatGPT provided on the other plots are actually based on the data - or if they are just 'likely conclusions' given the context.

Below is the detailed conversation. Note how ChatGPT again struggles with basing its conclusions on the actual data, instead generating arbitrary correlation coefficients. Importantly, this should not detract from the quality and usefulness of the actual plot produced (which is valid and easily replicated based on the code). However, it stresses again the need for caution when asking ChatGPT to interpret its own results.

Below is the full conversation. ChatGPT practically creates this in 'one go', except for the zoom level of the map.

Below is the conversation showing how ChatGPT tries to solve the task - but fails with the DBSCAN and KD-Tree approaches:

Below is the detailed conversation. This is a relatively rare example of ChatGPT making an erroneous assumption in the Python code, specifically assuming that the time between observations is always 5 seconds (an extrapolation it makes based on part of the data).

One thing worth noting is that GPT4 struggles with adding e.g. city names, addresses or zip codes to each area. This is because it does not have access to geocoding packages and it is reluctant to use potentially outdated knowledge before the 2021 training data cutoff.

If pushed to provide a "best guess" as to the country of Area 1, it proposes Netherlands (the correct answer is Denmark, so it's not even close). However, this turns out to be an issue with GPT4 using only a single decimal in the latitude and longitude values.

After requesting that it leverages the full precision GPS coordinates, GPT4 does manage to map 8 of 10 areas to Denmark (Aarhus), which is correct - but fails to map two areas, specifically Denmark (Ebeltoft) and Austria (Kirchberg). For this reason we showcase the map without the partial geocoding result.

We expect Code Interpreter will at some point be able to fetch Python packages online, which will enable it to solve this type of task with ease.

This conversation is fairly iterative with multiple adjustment prompts added in order to arrive at the final result.

In particular, we provide a few styling prompts and ask GPT4 to filter out events where ImuValid or SpeedValid equal 0. GPT4 struggles with labelling each event with a consistent 'event index' across all three outputs, but succeeds with some guidance. The results of one of the outputs is below (showing the event plot).

As evident from the results of the first prompt, many of the plots include missing observations because SpeedKMH is zero during part of the dataset. This is expected as the CANedge is configured to only record data when Speed exceeds zero. It also means that a lot of the acceleration events reflect the car starting up after a stand still.

To fix this, we make the follow-up prompt shown in 13 / 20 below. This follow-up prompt is non-trivial because of the iterative nature: ChatGPT has to both identify the top 10 acceleration events - while also ensuring that each event has 121 observations. It tries a few iterative approaches until it eventually decides on a rolling window methodology that achieves the task with sufficient efficiency.

As shown in the line charts, this method ensures in 9 of 10 plots that there is also 60 seconds of data before/after the actual event, even though that is not in principle a requirement in the initial filtration that GPT4 produces. As evident, one plot (Event 5) thus also lacks some data after the event, but seemingly makes the cut because it has sufficient data before the event to fulfil the requirement of 121 rolling observations.

After some minor styling prompts, we arrive at the final output - showing the top 10 harsh braking events along with the surrounding speed data.

ChatGPT manages to perform the fairly complex trip clustering with ease and also provides summary statistics of the trips.

Below is the workflow for generating the histogram. Note how ChatGPT wrongfully concludes that Martin is better at regenerative braking, which appears to contrast the results from the plot. More on this in the next section.

Below is the full workflow for the accumulated power consumption analysis.

As evident from the chat extract shown, ChatGPT appears to swap the names of the drivers when asked to provide a conclusion. Additionally, it only fixes this upon being informed of the error.

Interestingly, ChatGPT makes the same erroneous conclusion when reviewing the previous power consumption histograms, claiming that Martin was driving more efficiently. In fact, it even makes the same wrong conclusion when based on the extremely clear-cut battery consumption plot - generating new hallucinated consumption averages as part of its response.

As discussed previously, this appears to reflect GPT4 defaulting to its training data for conclusions, thus hallucinating 'likely' values for each driver instead of relying on the factual results it just produced.

If that is the case, then it is curious that ChatGPT consistently believes Martin is the better driver across sessions. Maybe this reflects an intrinsic bias in ChatGPT towards thinking men are better than women at driving. Or, maybe it thinks that the prompter is Martin (which is correct) and thus finds it likely that a conclusion that favorises the prompter is preferred. Regardless, it's surprising how frequently ChatGPT ignores the factual results and instead goes for hallucination.

When asked why it fails, ChatGPT appears to suggest that it's not leveraging the 'hard data' it produced in previous responses, but rather does a visual inspection of the plot.

It is unclear to us if ChatGPT actually tries to extract the numbers from the plot. We tested this ability in isolation, uploading the battery consumption plot in a new session and asking ChatGPT to extract the numbers and derive the insights. It explains that it will do so using Optical Character Recognition (OCR) and proceeds to extract five numbers (19.0, 17.5, 15.4, 15.0, 12.5 - i.e. including the two intended values). It fails in this test to make the intended conclusion as well. Based on this it appears GPT4 has some level of competence for plot based number extraction, but it is too sporadic to be reliable. As such, GPT4 may be telling the truth when it claims that it tries using the data from the plot - but it could also be that the numbers it uses are completely made up.

Regardless, when ChatGPT actually re-does the calculation to fetch the 'hard numbers' it correctly concludes that Josefine is the better driver.

GPT4 will often 'drop out' of the session you're in, in which case it will lose access to the uploaded data. Rather than starting up a new chat session, the simplest fix is to reupload the relevant data and ask it to retry.

GPT4 often hallucinates which variables should be used in a calculation or visualisation. It'll typically realise the mistake and prompt you to inform it about the correct variable name, but to save time you can inform it upfront.

If you know programming, it's useful to open the code produced by GPT4 in each step. Sometimes there are some odd decisions made by GPT4 in developing the code. In addition, GPT4 often remakes the code from scratch upon e.g. new attempts - so it can be difficult to keep track at times. We find it useful to request GPT4 to output the full consolidated code when relevant. If you do this, make sure to also ask it to output the result of the code in the same prompt. You can then save this and run it locally in case you need to reproduce the exact results later.

While GPT4 is fast, you quickly get lost in the rabbit hole and hours can pass. To minimise this, be very clear on what you need to ask before you start prompting. In fact, we recommend writing down your full initial prompt for each question, ensuring you detail it to the extent you believe necessary to provide GPT4 with a chance to 'get it right' in the first go.

In our tests, ChatGPT will struggle with large complex prompts. You'll notice that we therefore frequently use the phrase "Let's think step-by-step", which has been shown to enable the large language model (LLM) to more easily handle complex tasks. Without this phrase, ChatGPT tends to proceed more iteratively and plunge into the task. With the phrase, it typically outlines each step clearly up-front as a "plan" - and then proceeds to execute it, resulting in far fewer mistakes.

If you're going to use ChatGPT for similar analyses, you may want to sanitise your data up front. Specifically, you may consider removing GPS data from your dataset before uploading, as this will in practical terms make your data anonymous and may also be required for GDPR or CCPA purposes. This can be done easily via the asammdf GUI filters when exporting to CSV - or by using our Python API to filter out specific signals.

You'll often need to re-upload your data e.g. if a session expires. To speed this up you can zip the CSV file to reduce size by 80%+. This also allows you to easily upload multiple CSV files and/or larger CSV files than the normal file limit.

In many of the examples in this article we deliberately challenge ChatGPT by providing extensive prompts with multiple sub tasks and multiple outputs. In practice, it is often simpler to take it step by step, e.g. asking ChatGPT to produce one output at a time and wait for feedback before proceeding.

If you know certain variables need to be calculated for use in many prompts, do this in advance. This helps you get more consistent results between prompts and you avoid the risk of ChatGPT occasionally calculating a new variable incorrectly. Similarly, consider what sampling frequency and data structure minimises the effort from ChatGPT in order to perform your required analyses.

In our experience, working with GPT4 plus Code Interpreter is like having a junior analyst that produces output at an inhuman rate. Yet, like most junior analysts you'll need to provide clear guidance up front and diligently review any analyses produced.

This does not mean that you need to know Python code - but you should ideally have an understanding of how to work with time series data to get the most out of this tool.

Side comment

As we wrote the above tip, we wanted to exemplify this. Specifically, we believed ChatGPT would not proactively remove 'charging events' when calculating the 'consumed' state of charge over time. However, for the sake of good order we checked this - and we were wrong ...