Parsing Problem on compressed EEG


#1

Hello Farough and other developers.

I have a question on parsing compressed EEG package as well.

Below is one sets of data I recorded from muse between two synchronization package ([B]NOTE: in Decimal[/B]), for better reading I manually put enter in this post

255 255 170 85 – here is synchronization
224 185 45 97 197 122 – here is uncompressed EEG

192 7 22 90 104 129 – here is first compressed EEG
1 70 – here is length
108 38 150 180 80 22 150 104 83 22 80 204 22 243 193 216 190 86 70 97 128 22 11 122 217 44 95 11 31 81 192 154 233 54 101 34 170 194 17 138 80

192 6 22 90 40 129 – here is second compressed EEG
1 73
93 232 185 151 182 85 70 213 92 24 158 45 178 152 197 152 29 231 115 179 108 49 108 174 190 41 128 226 118 67 55 184 110 52 133 227 10 61 74 133 138 128

192 7 38 89 104 129 – here is third compressed EEG
1 86
154 19 197 148 38 210 80 170 219 9 159 137 121 79 146 13 105 54 170 225 101 131 191 114 226 184 59 53 76 171 18 176 47 132 39 20 160 218 50 117 94 201 52

Ignore the Uncompressed package.
I quite sure that I get the Medians from each Compressed EEG package is correct.
For example:
First Compressed EEG package:
Median1 is 7
Median2 is 5
Median3 is 5
Median4 is 5
Then I parse Quotient, remainder, Sign

median of 7 expect to see 0 = 00, 1 = 010, 2 = 011, 3 = 100, 4 = 101, 5 = 110, 6 = 111.
01101/100 00/100110/ 100101/10 1011/0100 0/101000/0 000/10110 1/00101/10 0110/1000 0/101001/1 0001/0110 0/101000/

median of 5 expect to see 0 = 00, 1 = 01, 2 = 10, 3 = 110 and 4 = 111.
0 1100/1100 00/0101/10 1111/0011/ 110000/01 110/11000 1/01111/10 010/10110 0/10001/10 011/0000/1 10000/000 0/

median of 5 expect to see 0 = 00, 1 = 01, 2 = 10, 3 = 110 and 4 = 111.
0010/110 000/0101/1 01111/010 1/101100/1 0010/1100 01/01111/1 0000/1011 00/01111/1 0101/0001/ 110000/00 10/

median of 5 expect to see 0 = 00, 1 = 01, 2 = 10, 3 = 110 and 4 = 111.
01101/0 1110/1001 0/01101/10 011/0010/1 0010/0010/ 10101/010 1/10000/10 000/10001/ 10001/010 0/10100/[B]00[/B]

I can read 64 sample deltas from above, but the problem is I didn’t fully read all bytes, 256+70 = 326 / 8 = 41 bytes (from bit length section: 1 70 [B]in decimal[/B]), there are two bits I didn’t use. The last two 0s. I hope you can see my problem.
And this problem seems happened to all my compressed EEG package, sometime it might left one 0 or more, which i didn’t find it out yet.

I am reading in the raw data from Muse i[B]n decimal[/B] instead of [B]hex[/B], is this could be an issue that cause this strange parsing? I had read the latest post about parsing from “runner’s” post (http://forum.choosemuse.com/forum/developer-forum/5030-problem-with-parsing-compressed-eeg-packets?_=1425318957890) and “zhuang’s” post (http://forum.choosemuse.com/forum/de...to-linux/page2). They are all reading raw data in Hex. I saw they can use all data and no left bit(s).

Could you help me to clarify this issue If it is a wrong parsing, please?

Many thanks

Sean

Here is my second uncompressed eeg package medians:
Second Compressed EEG package:

Median1 is 6
Median2 is 5
Median3 is 5
Median4 is 4

The last uncompressed EEG byte in decimal is 128, convert to binary will be 1000 0000 , the same happen here as well.

Just in case I put my parse data of the third package as well.
Third Compressed EEG package:

Median1 is 7
Median2 is 9
Median3 is 5
Median4 is 5

Median1 = 7 expect to see 0 = 00, 1 = 010, 2 = 011, 3 = 100, 4 = 101, 5 = 110, 6 = 111.
100110/10 000/10011 1/10001/01 100/10100 0/01001/10 1101/0010 0/101000/0 1010/1010 11/01101/1 0000/1001 10/01111/

Median1 = 9 expect to see 0 = 000, 1 = 001, 2 = 010, 3 = 011, 4 = 100, 5 = 101, 6 = 110, 7 = 1110, 8 = 1111
1 100010/01 011/11001 01/00111/1 100100/10 0000/1101 011/01001/ 00110/110 1010/1010 11/100001/ 01100/101 100/00011/ 101111/

Median1 = 5 expect to see 0 = 00, 1 = 01, 2 = 10, 3 = 110 and 4 = 111
11 01110/010 1/110001/0 101/11000 0/01110/11 0011/0101/ 0100/1100 10/10101/1 0001/0010/ 101100/00 00/101111/ 10000/100 00/10011/1 0001/0100/ 10100/000 1/101101/0 001/10010/ 01110/101 01/01111/0 1100/1001 0/01101/[B]00[/B]
Over here, the last two 0s didn’t use as well.


#2

Hi Sean,

I think the issue here is just a small misunderstanding. I haven’t gone through all your examples, which I can do if this is incorrect. But it appears your misunderstanding the meaning of the length.

So the length bits in the first example, 1 and 70, means you have 256 + 70 = 326 bits of data in your payload. Meaning 40.75 bytes, knowing that we can’t actually send 0.75 bytes of data for your payload, we need to round it up to 41. This means the last two bits will not be used as they are just added to the payload to make the packet fit series of bytes to allow consistent headers for the data stream.

Extrapolating further, 256 + 73 = 329 / 8 -> 41.125 meaning the last 7 bits are not relevant to the packet. (256+86)/8 = 42.75 bytes. Again 43 bytes sent, final 2 bits are unused.

Hopefully that clarifies what you’re seeing. I will analyze the bits closer for you if you still thing there is an issue.


#3

Hi SeanyP,

I don´t see any problem, unused bits is are expected here because the lenght of the payload is telling you that it’s not an exact multiple of 8 (so you need an extra byte to acomodate all the needed bits) and the extra bits should be ignored.

1 - 70 = 256 + 70 = 326
326 / 8 = 40.75 (that´s why you need 41 bytes)
40 bytes gives you 320 bits. From the last byte you take the first [B]6[/B] bits and ignore the last [B]two[/B] bits.

For the second uncompressed packet:
1 - 73 = 256 + 73 = 32[B]9[/B]
41 bytes gives 328 bits (so you need 42 bytes). From the last byte take just [B]1[/B] bit and ignore the last [B]7[/B] bits.
and so on …

If you are parsing the quotients, remainders and signs correctly everything will be OK.

Reading HEX instead of DECIMAL just make easy to see the binary numbers in my oppinion.


#4

Hi Farough and Eduardo

Thanks very much for both of your explanations, I understand it now.

AND Sorry for the late reply. I just want to do a bit more on my parsing and ask some more questions rather than just reply a “thank you”.

Now I am getting the 64 sample deltas.

Below is the actual delta values from the CH1 and CH2 of First compressed EEG package that giving in my previous post :

// After parsing, compute actual sample values:
// deltas = (quotient * median + remainder) * sign * quantization

median of 7
Quantisation1 = 16

[B]0 5 1[/B] / delta 1 = (07+5)116 = 90[B], [/B]
[B]1 0 -1 / [/B]delta 2 = (1
7+0)-116 = -9[B], [/B]
[B]1 2 -1 / [/B]delta 3 = (17+2)-116 = -144[B], [/B]
[B]1 1 1 / [/B]delta 4 = (1
7+1)116 = 128[B], [/B]
[B]1 4 1 / [/B]delta 5 = (17+4)116 = 176[B], [/B]
[B]0 3 -1 / [/B]delta 6 = (0
7+3)-116 = -48[B], [/B]
[B]1 3 -1 / [/B]delta 7 = (17+3)-116 = -160[B], [/B]
[B]0 0 -1 / [/B]delta 8 = (0
7+0)-116 = -16[B], [/B]
[B]1 5 1 / [/B]delta 9 = (17+5)116 = 192[B], [/B]
[B]0 1 1 / [/B]delta 10 = (0
7+1)116 = 16[B], [/B]
[B]1 2 -1 / [/B]delta 11 = (17+2)-116 = -144[B], [/B]
[B]1 0 -1 / [/B]delta 12 = (1
7+0)-116 = -112[B], [/B]
[B]1 3 1 / [/B]delta 13 = (17+3)116 = 160[B], [/B]
[B]1 0 1 / [/B]delta 14 = (1
7+0)116 = 122[B], [/B]
[B]0 5 -1 / [/B]delta 15 = (07+5)-116 = -80[B], [/B]
[B]1 3 -[/B]1 [B]/ [/B]delta 16 = (1
7+3)116 = 160

median of 5
Quantisation2 = 64
[B]0 3 -1[/B] [B]/ [/B]delta 1 = (05+3)-164 = -192[B], [/B]
[B]2 0 -1[/B] [B]/ [/B]delta 2 = (2
5+0)-164 = -640[B], [/B]
[B]0 2 1[/B] [B]/ [/B]delta 3 = (05+2)164 = 128[B], [/B]
[B]1 4 1[/B] [B]/ [/B]delta 4 = (1
5+4)164 = 576[B], [/B]
[B]0 1 1[/B] [B]/ [/B]delta 5 = (05+1)164 = 64[B], [/B]
[B]2 0 -1[/B] [B]/ [/B]delta 6 = (2
5+0)-164 = -640[B], [/B]
[B]0 4 -1[/B] [B]/ [/B]delta 7 = (05+4)-164 = -256[B], [/B]
[B]2 0 1[/B] [B]/ [/B]delta 8 = (2
5+0)164 = 640[B], [/B]
[B]0 4 1[/B] [B]/ [/B]delta 9 = (05+4)164 = 256[B], [/B]
[B]1 1 -1[/B] [B]/ [/B]delta 10 = (1
5+1)-164 = -384[B], [/B]
[B]1 3 -1[/B] [B]/ [/B]delta 11 = (15+3)-164 = -576[B], [/B]
[B]1 0 1[/B] [B]/ [/B]delta 12 = (1
5+0)164 = 320[B], [/B]
[B]1 1 1[/B] [B]/ [/B]delta 13 = (15+1)164 = 384[B], [/B]
[B]0 0 -1[/B] [B]/ [/B]delta 14 = (0
5+0)-164 = -64[B], [/B]
[B]2 0 -1[/B] [B]/ [/B]delta 15 = (25+0)-164 = -640[B], [/B]
[B]0 0 -1[/B] [B]/ [/B]delta 16 = (0
5+0)-164 = -64
……

Following the document on the Muse Compressed EEG package, it says
[B]“ deltas = (quotient * median + remainder) * sign * quantization[/B]
[B]for each quotient in set[/B]
[B] Add delta to previous sample to get next sample[/B]
[B]The final sample should be stored as the starting values for the next compressed EEG packet.”[/B]

What does it mean by “for each quotient in set
Add delta to previous sample to get next sample “ ?

I am already calculate each deltas above, does the next delta require to add previous one, which on my example above the detla 1 = 90, delta 2 = -9, but instead of -9 the value is 90-9 = 91 and delta 3 = 91 + (-144) = -53 but not -144 ?? I am confusing about this sentence.

And another question is about the “Differencing Method”, I didn’t use this method yet. But I don’t know if I should use it and where should I use it? It seems like I have to use it here now, after I get 64 sample deltas.

Could you give a bit clear explanation on this method please? If there is an example, it will be really helpful.

Thanks
Sean


#5

Hi Sean,

I apologize if the wording seems cryptic or ambiguous.

Basic idea here is the packet itself contains 64 deltas. Where a delta is a different between one sample and it’s immediate neighbour.

For example:


Let’s say your initial value is 500. Where this value is defined by the first packet you get which is an uncompressed EEG. This is denoted by the header byte 0xE0.

In this example we will look at one channel and assume the first packet said channel 1 was 500.
If your subsequent compression packet then has 16 differences as followed (-3, +5, +2, -6, +4, -1, +6, +3, -3, +5, +2, -6, +4, -1, +6, +3), again this is just an example.

You would find the next 16 values for channel one by applying the deltas one at a time.
Here as follows:

500, 497, 502, 504, 498, 502, 501, 507, 510, 507, 512, 514, 508, 512, 511, 517, 520.


Please note that in your example your samples are quantized, this should only be common if no user is wearing the headband.

For each delta in the set I’ve added it to the previous EEG value reaching 16 new values. Your following packet would then append to this set. Allowing each next packet to give you 16 new EEG values for each channel a total of 64 next values.

I’ve only laid out channel one here, but you would complete the same operation for each channel.

Please keep in mind adding the delta’s means you considered a delta sign bit of 1 to mean a negative sign and a sign bit of 0 to mean a positive sign.
As below: [TABLE=“border: 0, cellpadding: 0, cellspacing: 0, width: 188”]
[TR]
[TD][B]Sign Bit[/B][/TD]
[TD]1[/TD]
[TD]0[/TD]
[/TR]
[TR]
[TD][B]Sign[/B][/TD]
[TD]Negative[/TD]
[TD]Positive[/TD]
[/TR]
[/TABLE]

Hope that helps clarify.


#6

Hi Farough

I understand it now. Thanks for the example.

You mentioned that “in your example your samples are quantized, this should only be common if no user is wearing the headband.” I actually forgot how I record the raw data, you must right that I didn’t wear the headband and record those data. And I will record few sets of data again when I am wearing the Muse properly. Thanks for point out this issue for me.

So, Can I say If I am wearing the Muse properly and record data, then the difference will be small, not like the one I had on previous post, each difference are huge number?
like this:
[B]0 5 1[/B] / diff 1 = (07+5)116 = 90[B], [/B]
[B]1 0 -1 / [/B]diff 2 = (1
7+0)-116 = -9[B], [/B]
[B]1 2 -1 / [/B]diff 3 = (17+2)-116 = -144[B], [/B]
[B]1 1 1 / [/B]diff 4 = (1
7+1)116 = 128[B], [/B]
[B]1 4 1 / [/B]diff 5 = (17+4)116 = 176[B], [/B]
[B]0 3 -1 / [/B]diff 6 = (0
7+3)-116 = -48[B], [/B]
[B]1 3 -1 / [/B]diff 7 = (17+3)-116 = -160[B], [/B]
[B]0 0 -1 / [/B]diff 8 = (0
7+0)-116 = -16[B], [/B]
[B]1 5 1 / [/B]diff 9 = (17+5)116 = 192[B], [/B]
[B]0 1 1 / [/B]diff 10 = (0
7+1)116 = 16[B], [/B]
[B]1 2 -1 / [/B]diff 11 = (17+2)-116 = -144[B], [/B]
[B]1 0 -1 / [/B]diff 12 = (1
7+0)-116 = -112[B], [/B]
[B]1 3 1 / [/B]diff 13 = (17+3)116 = 160[B], [/B]
[B]1 0 1 / [/B]diff 14 = (1
7+0)116 = 122[B], [/B]
[B]0 5 -1 / [/B]diff 15 = (07+5)-116 = -80[B], [/B]
[B]1 3 -[/B]1 [B]/ [/B]diff 16 = (1
7+3)116 = 160

I saw runner’s last post about his parsing, and he finally got the difference values most are single digit.
like this: [TABLE=“class: text_table”]
[TR]
[TD=“width: 25%”][SIZE=11px]3 0 -1 -> -15
5 0 -1 -> -25
2 0 1 -> 10
0 4 -1 -> -4
0 1 1 -> 1
4 3 1 -> 23
0 4 1 -> 4
0 3 -1 -> -3
0 0 1 -> 0
7 3 1 -> 38
1 0 1 -> 5
1 2 -1 -> -7
3 0 1 -> 15
2 3 1 -> 13
0 1 1 -> 1
0 2 -1 -> -2 [/SIZE][/TD]
[TD=“width: 25%”][SIZE=11px]4 0 1 -> 8
3 0 1 -> 6
2 1 -1 -> -5
2 0 -1 -> -4
2 1 1 -> 5
0 0 -1 -> 0
1 0 1 -> 2
1 0 -1 -> -2
1 0 1 -> 2
1 1 -1 -> -3
1 0 1 -> 2
1 0 -1 -> -2
0 0 -1 -> 0
0 0 1 -> 0
1 1 1 -> 3
1 0 -1 -> -2 [/SIZE][/TD]
[TD=“width: 25%”][SIZE=11px]1 1 1 -> 8
2 2 -1 -> -16
0 0 1 -> 0
1 1 -1 -> -8
0 3 -1 -> -3
1 0 1 -> 7
0 6 1 -> 6
0 5 1 -> 5
0 3 -1 -> -3
0 6 1 -> 6
0 4 -1 -> -4
3 1 -1 -> -22
2 2 1 -> 16
1 2 1 -> 9
1 5 1 -> 12
2 0 -1 -> -14 [/SIZE][/TD]
[TD=“width: 25%”][SIZE=11px]2 0 -1 -> -10
0 2 -1 -> -2
1 2 1 -> 7
0 0 -1 -> 0
1 4 -1 -> -9
0 0 -1 -> 0
1 2 1 -> 7
1 0 1 -> 5
1 1 -1 -> -6
0 2 -1 -> -2
1 0 1 -> 5
1 3 1 -> 8
0 0 1 -> 0
1 1 -1 -> -6
0 0 1 -> 0
1 3 1 -> 8[/SIZE][/TD]
[/TR]
[/TABLE]

Please, May I ask what exactly mean " are quantized" in that sentence? Should I understand it as random data, since the headband wasn’t worn when data been recorded?

And One more question, when the special case Median = 1 happens, the remainder will always be 0 (only 1 bit), am I right? and for Median = 2, we are expecting to see 0 or 1 (also only 1 bit) ? I saw the previous post when you answer zhuang’s question, he figured out this special case and he manually set the value for those cases. But you didn’t mention what was the remainder when Median = 1. I just want to make sure the value I said is correct. Please confirm it is correct or not?

Thanks
​Sean


#7

Hi Sean,

Good to hear.

Let me explain the quantization a bit better.

You won’t necessarily see it only when it is not being worn. Basically what happens is that when the deltas between samples are too large the compression algorithm would need a lot of bandwidth to send the packets. Our solution to keeping the bit rate down is to start to quantize the data and make it lossy at the benefit of reducing the required bandwidth.

Therefore, if you’re wearing the headband poorly with low contact to your skin, you can expect to see some level of quantization.

The justification here if also if your data has extremely large swings then the brainwaves may not be possible to detect anyway, so there’s no reason to send large packets that use a lot of bluetooth bandwidth. By quantizing we don’t need to send as much bits to represent our data. We effectively divide by the level of quantization.

You’re correct about Median’s of 1. I thought I mentioned that in Zhuang’s Question. Median of 1 will always be 0 with 1 bit, median of 2 will also have 1 bit, 0 or 1.