This is pretty interesting:
The rate at which an organism utilizes energy, measured in Kcal/ hr or other units of energy per time is metabolic rate, and if measured in Kcal/kg - hour or similar units of energy/mass-time can be considered specific metabolic rate. The rate at rest is the basal metabolic rate (BMR), defined for humans to be the metabolic rate that is found after a person has fasted for 12 hours, slept for at least 8 hours, and has been resting quietly for at least half an hour. Muscular activity will, of course, increase the metabolic rate. Since the conditions for BMR are difficult to achieve, we will really be measuring Resting Metabolic Rate (RMR), which does not require the fasting and half hour resting, and is expected to be a little higher.
Metabolic rate can be measured by measuring the temperature change of a closed container containing a metabolizing organism, since the temperature change and heat production are just related by a proportionality constant – the heat capacity. This is called direct calorimetry. It is accurate, but requires very specialized equipment so it is inconvenient, especially for large animals. Since living organisms use molecular oxygen only for cellular respiration, the rate of oxygen consumption is directly related to the rate of ATP production. Also, since oxygen is not given off as a by-product of other cellular reactions (except, of course, photosynthesis), it must be obtained from the environment. Because oxygen consumption from the atmosphere can be easily and accurately measured without resorting to complicated technical analytical procedures, metabolic rates of terrestrial organisms can be obtained. In this lab we will take advantage of the stoichiometric relationships between energy production and oxygen consumption, and measure oxygen consumption. This is called indirect calorimetry, which has several variants.
I guess a good way to determine if someone 'lost muscle' would to set themself up using indirect calorimetry PRIOR to dieting....then do it again quite a ways INTO the diet when strength loss starts to occur. Being that skeletal muscle is the most metabolic tissue in the human body (up to 8x more than fat cells) you could say with some degree of certainty that some of the metabolic change could have come from 'muscle loss'. Again, not saying it would be an absolute....just a possible scenario.