The best place to start learning about brain waves is probably the wikipedia article on Alpha waves: https://en.wikipedia.org/wiki/Alpha_wave
From that page there are many many links to other articles.
So far as identifying relaxation goes, the best place to start would be the research paper “My Virtual Dream: Collective Neurofeedback in an Immersive Art Environment” by Natasha Kovacevic, Petra Ritter, William Tays, Sylvain Moreno, Anthony Randal McIntosh: http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0130129
According to this paper, relaxation can be determined as an increase in relative spectral power of Alpha. Doing a quick Google search for “EEG relaxation” will bring up many similar papers mentioning Alpha. You should note however that none of these theories are proven. Currently neuroscience is very much in it’s infancy; there is a lot we do not understand and there is no simple guide which says frequency X means Feeling Y.
Also bear in mind that it has taken Interaxon many years and a team of neuroscientists to come up with their secret algorithm for their Calm app.
Muse Monitor was designed to give easy access to EEG data for neuroscience research students, to further mankind’s understanding of the field.