PLEASE HELP ME get Muse working with NeuroMage for the NeuroGaming conference


#1

So this is the gameplay video, just moved onto an Android tablet… runs pretty well on anything Samsung S4 or better speed wise.

In the game you must meditate to learn your spells… the crystal ball shows the current meditation level and the crystals show your progress learning the spells.

I’d REALLY like to have this on the Muse and if I can use calm / concentrate then it will exactly mirror the Neurosky’s meditation / attention … should be easy once I have that data coming in. It already includes multi-player, minion battles, and tower defense elements. Lots more to show than just this short video.

I’ve got the SDK into Unity but MuseState always returns 0.

Here is the log:

05-02 15:42:50.871: D/Raydium_ts_main(561): ## Report rate in 10 seconds =1627
05-02 15:42:50.959: W/Unity(8432): callGetStat in muse
05-02 15:42:50.959: W/Unity(8432):
05-02 15:42:50.959: W/Unity(8432): (Filename: ./artifacts/generated/common/runtime/UnityEngineDebug.gen.cpp Line: 56)
05-02 15:42:50.959: W/Unity(8432): We have currentActivity object… no errrors in callGetStat
05-02 15:42:50.959: W/Unity(8432):
05-02 15:42:50.959: W/Unity(8432): (Filename: ./artifacts/generated/common/runtime/UnityEngineDebug.gen.cpp Line: 56)
05-02 15:42:50.960: D/BluetoothAdapterService(1101097728)(1582): Get Bonded Devices being called
05-02 15:42:50.960: D/BluetoothAdapterProperties(1582): getBondedDevices: length=2
05-02 15:42:50.962: W/Unity(8432): MUSE — STATE=0
05-02 15:42:50.962: W/Unity(8432):
05-02 15:42:50.962: W/Unity(8432): (Filename: ./artifacts/generated/common/runtime/UnityEngineDebug.gen.cpp Line: 56)
05-02 15:42:51.126: W/Unity(8432): callGetStat in muse
05-02 15:42:51.126: W/Unity(8432):
05-02 15:42:51.126: W/Unity(8432): (Filename: ./artifacts/generated/common/runtime/UnityEngineDebug.gen.cpp Line: 56)
05-02 15:42:51.126: W/Unity(8432): We have currentActivity object… no errrors in callGetStat
05-02 15:42:51.126: W/Unity(8432):
05-02 15:42:51.126: W/Unity(8432): (Filename: ./artifacts/generated/common/runtime/UnityEngineDebug.gen.cpp Line: 56)
05-02 15:42:51.128: D/BluetoothAdapterService(1101097728)(1582): Get Bonded Devices being called
05-02 15:42:51.128: D/BluetoothAdapterProperties(1582): getBondedDevices: length=2
05-02 15:42:51.130: W/Unity(8432): MUSE — STATE=0
05-02 15:42:51.130: W/Unity(8432):
05-02 15:42:51.130: W/Unity(8432): (Filename: ./artifacts/generated/common/runtime/UnityEngineDebug.gen.cpp Line: 56)

I have a debug as you can see above where
I write out “We have currentActivity object… no errrors in callGetStat”… So it is finding the java library fine it appears.

It’s coming down to just three lines of code I’m using to test if I’m connected…

AndroidJavaClass unity = new AndroidJavaClass (“com.unity3d.player.UnityPlayer”);
AndroidJavaObject currentActivity = unity.GetStatic<AndroidJavaObject> (“currentActivity”);
musestate = currentActivity.Call <int> (“ConnectMuse”);

Am I missing something to actually find the Muse device itself?

I feel like I’m SOOOOO close. All I need is the experimental/meditation/calm data and I can punch it right into the game. But time it ticking!

Anyone want to try and help me figure this out?

Greg Moss
NeuroMage.net


#2

Hi Greg,

Congratulations for your persistence … as you said in another post, like most of us in this forum, I know nothing about Unity, so just trying to guess …

Good luck, I wish you great success with your project !

BTW: For awhile I don’t trust too much in the experimental “mellow / concentration” indicators, it’s also my guess that they tried to implement something similar with what the Calm app uses, but it doesn’t seem reliable yet … to me :slight_smile:

Eduardo


#3

Hi Greg,

From your post I’m a bit uncertain about what could be causing the issue. However I can outline the system we used for Android in the past and perhaps you can use a similar system.

Basically we created a JSON in Android and then Unity would pull that JSON. Android code, like in the SDK, handled all the muse connection code.

[B]Android[/B] [B]Code[/B] looked like this:

/**

  • Retrieves all of the data that is currently stored within the buffer,
  • clearing any data marked to be cleared.
  • @return JSON object containing all of the buffered data.
    */
    public synchronized JSONObject flushBufferedDataAsJson() {
    JSONObject state = new JSONObject();
    try {
    state.put(“noise”, this.noise);
    state.put(“battery_charge”, this.batteryCharge);
    state.put(“battery_voltage_fuel_gauge”,
    this.batteryVoltageFuelGauge);
    state.put(“battery_voltage_adc”, this.batteryVoltageAdc);
    state.put(“temperature”, this.temperature);
    state.put(“data_incoming”, this.dataIncoming);
    state.put(“is_good_overall”, this.isGoodOverall);

JSONArray rawData = new JSONArray();
for (float[] rawSet : this.rawData) {
JSONArray jsonRawSet = new JSONArray();
for (float val : rawSet)
jsonRawSet.put(val);
rawData.put(jsonRawSet);
}
this.rawData.clear();
state.put(“raw_data”, rawData);

state.put(“is_config_updated”, this.museConfigChange);
if (this.museConfigChange) {
JSONObject configuration = new JSONObject();
if (this.muse.isPresent()
&& this.muse.get().currentConnectionStatus() == ConnectionStatus.CONNECTED) {
configuration.put(“bluetooth_name”, this.muse.get()
.getBluetoothName());
configuration.put(“mac_address”, this.muse.get()
.getMacAddress());
configuration.put(“version”, this.muse.get().version()
.toString());
configuration.put(“serial”, this.muse.get().configuration().getSerialNumber()) ;
state.put(“battery_charge”, this.muse.get()
.configuration().getBatteryPercentRemaining());
} else {
configuration.put(“bluetooth_name”, “”);
configuration.put(“mac_address”, “”);
configuration.put(“version”, “”);
state.put(“battery_charge”, 0);
}
state.put(“configuration”, configuration);
this.museConfigChange = false;
}

JSONArray noisePerChannel = new JSONArray();
for (int[] noiseSet : this.noisePerChannel) {
JSONArray jsonNoiseSet = new JSONArray();
for (int val : noiseSet)
jsonNoiseSet.put(val);
noisePerChannel.put(jsonNoiseSet);
}
this.noisePerChannel.clear();
state.put(“noise_per_channel”, noisePerChannel);
} catch (JSONException e) {
e.printStackTrace();
}
return state;
}

And then the [B]Unity[/B] [B]Code[/B] looked like this:

// Update is called once per frame
void Update ()
{
if (!hasInit) {
initialize ();
hasInit = true;
}
#if UNITY_EDITOR
#elif UNITY_ANDROID
else
{
updateState(IXBasisManager.ixbasisDroidController. CallStatic<string>(“flushBufferedDataAsJson”));
}
#endif
}

#if UNITY_EDITOR
#elif UNITY_ANDROID
void updateState(string jsonState)
{
AndroidMuseStateModel state = JsonConvert.DeserializeObject<AndroidMuseStateMode l>(jsonState);

if (state.noise != null)
{
IXBasisManager.museController.SetMuseNoiseLevel(st ate.noise);
}

IXBasisManager.museController.SetMuseDataIncomingF lag(state.data_incoming);
IXBasisManager.museController.SetMuseDataRate(stat e.sample_rate.ToString());
IXBasisManager.museController.SetMuseGoodData(stat e.is_good_overall);

string battery = “”;
battery += state.battery_charge + “,”;
battery += state.battery_voltage_fuel_gauge + “,”;
battery += state.battery_voltage_adc + “,”;
battery += state.temperature;
IXBasisManager.museController.SetMuseBattery(batte ry);

foreach (float[] data in state.raw_data)
{
IXBasisManager.museController.SetEEGState(data);
}

foreach (int[] data in state.noise_per_channel)
{
IXBasisManager.museController.SetQuantization(data );
}

if (state.is_config_updated)
{
IXBasisManager.museController.SetMuseName(state.co nfiguration.bluetooth_name);
IXBasisManager.museController.SetMuseMAC(state.con figuration.mac_address);
IXBasisManager.museController.SetMuseFirmware(stat e.configuration.version);
IXBasisManager.museController.SetMuseSerialNumber( state.configuration.serial);
}
}
#endif

Of course you will want to modify your JSON to contain data you’re interested in using. This covers a lot of the data in the SDK, but not everything.

Note that we were buffering all the data until a new frame was presented so the Unity side didn’t receive any new data until it was ready to use it for the next frame. Pulling system.

I apologize this forum removed all the tabbing from the code so it looked a bit messy.

I hope that helps and I’m excited to see some coverage from Neuro Gaming. We will have our Developer Evangelist Tom Hobson down there for the conference. Hopefully he can either see it or help you more if you need it.

Good luck!

Matt


#4

I think that is just what I needed! Going to try putting it in now. I’m optimistic because I have the SDK in there… It just appears the connect method isn’t enough to make it happen by itself.

On the mellow / concentration data… I had made a post in another thread about that concern. I’ve already spent time looking at the data using the console and could tell it just doesn’t track all that well… So I was thinking about looking at extra theta/delta and less beta/alpha to better define “mellow”. I don’t really need concentration as the idea of the game as you can see in the video above is that you have to relax and stay calm while casting spells and making decisions about how to handle the battle. So you are already having a lot of ‘concentration’ to play the game. It is a “cool under pressure” concept and a good ‘mage’ can control their mind and reach deeper and deeper levels of relaxation. Eventually we do want to go beyond just the basic meditation / attention that are in Neurosky and look closer at theta and even delta for really hard spells.

Once I struggled to get the muse working in Unity and so many other things in the game to pull together I haven’t worried too much about exactly what data signals to use. It should be relatively easy to add additional data into the compilation… The game itself basically takes between 0 (least relaxed) and 1 (most relaxed). NeuroSky’s meditation composite is pretty darn good. After a few hours of practice you can really make it work. I was hoping once having Muse’s value in the game and some practice it may prove to be a useful score as well. I also questioned if I could really improve on Muse’s indicator at this point. That’s probably best left to people with far better computational / algorithmic skills than myself.

I’ll keep you guys posted.


#5

MUSE CONNECTED!!! Still a bit to go but that was the big first step! Many thanks for all the help!


#6

Data is coming in… what would be the activity call to get the mellow value? currentActivity.Call<float> (“mellow”);


#7

The only way to get mellow from libmuse on android is register a mellow listener and then do this (similar to all other packets):
@Override
public void receiveMuseDataPacket(MuseDataPacket p) {
switch (p.getPacketType()) {
case EEG:
updateEeg(p.getValues());
break;
case MELLOW:
double mellow = p.getValues().get(0);
// do smth with mellow
break;


#8

Hi Greg,

Have you had success with this process now?


#9

Also interested to hear how it went. Also, couldn’t find any video reports from this years conference, are there any?