Well lobsters actually come 3 different ways... (generically speaking...of Maine or Canadian lobsters which are the most common here in the states)
Soft shell
Medium Shell
and Hard Shelled
Hardshelled lobsters are the most "common"
However some companies cut corners and save money by selling medium shells as hard shelled lobsters...
Softshell not quite like the crab... has a soft shell you can press into... (usually during spawning or shedding of skin) a lobster with a non-hard shell will have the effects of exactly what you described, same is true for medium shell.
So when buying lobsters make sure its a hardshelled lobster. So you dont get as much loss in the meat when you cook them.
If your trying to save money... and its a size issue then you could do a medium shell... (for example when you see those Lobster specials on tuesdays... its like 12.99 for a lobster potatoes and other crap... and you ask yourself why is it so damn cheap... Now you know.
If you get a chance, you should read two awesome books about lobster.
The Secret Life of Lobsters by Trevor Corman?
and
The Lobster Coast by Colin Woodward.
The hard shell, soft shell issue is caused by the molting process. When a lobster molts, it absorbs seawater until it's shell literally bursts open, then it crawls out and forms a new shell, thus growing a size. The soft shell you speak of results from the lobster being caught too near the molting process.
The meat infill ratio you referred to is also a part of this molting process. Think of buying your kid a pair of shoes that's a size too big, because he's going to grow into them. Same with lobsters. They hold the extra seawater until the new shell hardens, then release it. The lobster now has room to grow in its new shell.
You won't find molts in the winter as this process is triggered by warm water.
The other problem with soft shells is they are weak from the molting process and don't handle shipping very well, which is why the vast majority of soft shell lobsters go into processing for tails and meat.
You might find it interesting that the industry standard is 4-6% of live lobsters die before the customer can use them. It's dangerous to eat a lobster that has died before butchering or cooking whole. A toxin begins to form as soon as the lobster dies, and if enough of this toxin is present, it can make you very ill, regardless of how much you cook it.