Understanding AurioTouch

I have been playing around with CoreAudio on iOS of late. The trouble with media API’s is that they are necessarily complex and CoreAudio is no exception.

While trying to figure out how to read data coming from the microphone and visually render the samples to the screen, I came across the aurioTouch example provided by Apple. It looked great..until I tried to work out what the code was doing!

There are so many aspects of the code that I struggled to make sense of, from arbitrary scaling factors to the odd bit of inline assembly, but here I will mention just one. In hindsight, it doesn’t seem so obscure now. But hindsight is a wonderful thing.

After having obtained a buffer full of PCM audio data, the following code is used to fill an array of values that is used to draw the data:

SInt8 *data_ptr = (SInt8 *)(ioData->mBuffers[0].mData);
for (int i=0; i<numFrames; i++)
{
    if ((i+drawBufferIdx) >= drawBufferLen)
    {
        cycleOscilloscopeLines();
        drawBufferIdx = -i;
    }

    drawBuffers[0][i + drawBufferIdx] = data_ptr[2];
    data_ptr += 4;
}

m_Buffers[0].mData contains an array of SInt32 values. These are PCM samples in 8.24 fixed-point format. This means that nominally, 8 bits of the 32 bits are used to contain the whole number part, and the remaining 24 bits are used to contain the fractional part.

I could not understand why the code was iterating through it using an SInt8 pointer and why, when the actual value was extracted, it was using data_ptr[2]. i.e It was using the third byte of the 32 bit (4 byte) 8.24 fixed point sample and chucking away the rest. I was so confused that I turned to stackoverflow for help. The answer given is spot on the money…but perhaps not all that clear if you are an idiot like me.

After printing out the binary contents of each sample I finally understood.

The code is using an SInt8 pointer because, at the end of the day, it is only interested in a single byte of data in each sample. Once this byte of data has been extracted, data_ptr is advanced by 4 bytes to move it to the beginning of the next complete sample (32 bit, 8.24 fixed point format)

The reason it extracts data_ptr[2] becomes apparent when you look at the binary. What I was failing to appreciate (a school boy error on my part) was that the samples are in little-endian format. This is what a typical sample might look like in memory:

data_ptr[0]     data_ptr[1]     data_ptr[2]    data_ptr[3]
----------------------------------------------------------
 01001100    |   01000000    |   11001111    |  11111111
----------------------------------------------------------

The data is little-endian, meaning the least significant byte has the lowest memory address, and conversely, the most significant byte has the highest memory address. In CoreAudio 8.24 fixed point LPCM data, the first (most significant) 8 bits is used to indicate the sign. They are either all set to zero or all set to one. The sample code ignores this and looks at the most significant byte of the remaining 24 bits…which is data_ptr[2]

It is safe to throw the rest away as it is of little consequence to the display of the signal; throwing the rest of the data away still gives you a ‘good enough’ representation of the sample.

Later on in the sample code (not shown above), this value is divided by 128 to give a value between -1 and 1. It is divided by 128 because an SInt8 can hold a value ranging from -128 to +127

Like I said, this is just one of many confusing bits of code in the sample app. CoreAudio is not for the feint hearted. If you are a novice, like me, then perhaps the aurioTouch sample is not the best place to start!

 

Guided Access in iOS 6

Improving fine motor control and having a blast

My son is autistic. Trying to explain what this means is very difficult. His autism means that at the age of four, among many many other things, he has no language, behaves very unpredictably and struggles with his fine motor skills. He is also the worlds biggest fan of Sound Blaster on the iPad.

A typical session finds him constantly distracted by three things:

1. The home button. It’s tactile and tempting and he just cant resist its constant allure.

2. Inadvertant use of multi-touch gestures or “fat fingers”. This always ends up throwing him out of the app.

3. On the iPhone, the Ad banner gets mistakenly hit. Again, taking him away from the action.

It is remarkable that for a boy who could not even understand his own name a few months ago, he is still able to navigate back to the app from any of these eventualities. This is because he knows that the home button will get him back to a known location, from which he can navigate back to the app.

One button.

Genius.

My wife is a teacher. She teaches small children of about 6 years of age and is just starting to use the iPad in the classroom. Trying to keep kids engaged on one task and ensuring they dont change all your settings when your back is turned is an issue all teaches and parents have to contend with.

With Apples latest update to iOS these are problems that can now be addressed. iOS 6, launching in just a couple of months, has a new feature called guided access. This lets you disable physical buttons, such as the home button, lock out certain areas of the screen, thus disabling input and even prohibit activation of the devices motion sensors.

Such features ensure that teachers, parents and supervisors have a way to keep a kids focus inside of the app…or even on just a specific part of an app. For my 8-bit fanboy son this means that we can prevent him being thrown out of his favorite app in all three of the situations outlined above.

No doubt this will also prove to be a fantastic feature for restaurants (menus), kiosks (tourist information), display-only setups (show rooms) and various other task centric scenarios.

Fo me, this means the iPad will continue to be a useful learning device for my son and hopefully my home button might last that little bit longer before giving up the ghost.