Acoular is a Python library that processes multichannel data (up to a few hundred channels) from acoustic measurements with a microphone array which is stored in an HDF5 file. This blog post explains how to convert data available in other formats into this file format. As examples for other file formats we will use both .csv (comma separated text files) and .mat (Matlab files).
To demonstrate how to import and convert the data, we first need to get some data. In our case we use data from Zenodo, where a 64 channel recording of a scene with three sources is available in a number different formats. We use Python's urllib for the download. Depending on your internet speed this may take a while:
import urllib.request
url = 'https://zenodo.org/record/5809069/files/'
for filename in ('three_sources.h5','three_sources.csv',
'three_sourcesv7.mat','three_sourcesv73.mat'):
urllib.request.urlretrieve(url+filename, filename)
Now we have the same data in four different formats: Acoular's HDF5, .csv, Matlab version <= 7 and Matlab versions >=7.3.
HDF5 format is an open all purpose numerical data container file format. Data objects inside HDF5 files are stored in tree-like structure comparable to files and folders in a file system. Lets open the file and explore this structure, which is very simple in this case.
We use the pytables library to access the file. This the very same library used by Acoular under the hood. Alternatively Acoular can also work with h5py.
You could also use an HDF5 file viewer with a GUI (e.g. HDFView).
import tables
h5file = tables.open_file('three_sources.h5', mode = 'r') # read only mode
h5file.root
In its root sits just one object (one 'child'), which is an EArray (extensible array). Lets inspect its properties:
h5file.root.time_data
We see that this array has the size of 51200 (samples) by 64 (channels). The values are stored as 32 bit float numbers. While less than the usual 64 bit, 32 bit accuracy is more than enough in this case and it saves file space. The data itself can be accessed just like for a numpy array. As an example, we read the first 10 samples of channel 47.
h5file.root.time_data[:10,47]
Along with the data itself, the object stores also some metadata ('attributes').
h5file.root.time_data.attrs
There is one custom attribute here, which is sample_freq. It specifies the sampling frequency. In our case 51200.0 Hz.
If we now have data in some other format that we want to use with Acoular, there are two options:
- We read that data and convert it into an HDF5 file that follows the specification explained. This is demonstrated in this blog post.
- We extend Acoular to read the other file format directly. This would mean to subclass the
TimeSamplesclass and requires some understanding of Acoular's code and working mechanism.
The first option shall now be demonstrated using .csv formatted data. Despite beeing extra inefficient this human-readable text format is widely used. The file contains the same number of floating point numbers separated by commas on each line. Some .csv files have also one or more header lines explaining the data contained in the file. In our case there are no header lines. There are multiple options how to read such file into Python. We are going to use Numpy for this. Be warned, the import of this (relatively small) 80 MByte file takes some time.
import numpy as np
datacsv = np.genfromtxt('three_sources.csv', delimiter=',', dtype='float32')
datacsv
Now the data is stored in the datacsv array. the next step is to create a new HFD5 file, store the data into that file and add the attribute for the sampling frequency.
h5filecsv = tables.open_file('three_sources_from_csv.h5', mode='w',
title='three_sources')
earraycsv = h5filecsv.create_earray('/', 'time_data', obj=datacsv)
display(earraycsv)
h5filecsv.root.time_data.set_attr('sample_freq',51200.0)
h5filecsv.close()
Just as before with the original HDF5 file we now have the data in new HDF5 file that could be used as data source for Acoular. There is one possible pitfall with this approach: the data is completely read into the computer memory before beeing stored into the HDF5 file. If the data is really huge, say hundreds of channels and some minutes of recording, it might not fit into the memory. In this case, a more sophisticated approach is needed, where chunks of data are read and stored consecutively. Because we use an EArray, this is possible, but we would have to modify the code.
As mentioned before, there are other options to read the .csv data. One that deserves to be mentioned here is Pandas which reads a lot of different data formats.
For some reason it is quite popular to store data in the format used by Matlab. However, it is important to know that despite the same extension (.mat), there are different formats. If we have any of the formats used prior to Matlab v7, then we can use Scipy to import this:
from scipy.io import loadmat
ans = loadmat('three_sourcesv7.mat')['ans']
datamat7 = np.array(ans, dtype='float32')
h5filemat7 = tables.open_file('three_sources_from_mat7.h5', mode='w',
title='three_sources')
earraymat7 = h5filemat7.create_earray('/', 'time_data', obj=datamat7)
display(earraymat7)
h5filemat7.root.time_data.set_attr('sample_freq',51200.0)
h5filemat7.close()
The format of .mat file from version 7.3 onwards is essentially an HDF5 file itself! It just uses another file name extension. Of course the internal structure is different from what Acoular is using. However, we can open it with pytables and read the data in it.
matfile73 = tables.open_file('three_sourcesv73.mat', mode = 'r')
# be aware of Matlab transposing the array here
datamat73 = np.array(matfile73.root.ans[:,:], dtype='float32').T
h5filemat73 = tables.open_file('three_sources_from_mat73.h5', mode='w',
title='three_sources')
earraymat73 = h5filemat73.create_earray('/', 'time_data', obj=datamat73)
display(earraymat73)
h5filemat73.root.time_data.set_attr('sample_freq',51200.0)
h5filemat73.close()
This blog post has demostrated how to import data from foreign formats into Acoular. It can also be used as a guide how to convert any other formats not explicitly mentioned here.