I've run into this too.
Strangely, I can delete 'text area' fields, but not 'checkbox' fields. This is particularly frustrating, because the most common mistake I seem to make while setting these up is forgetting to select a field type in the top box, which means it defaults to checkbox.
Since no one has replied, I guess I'll be going in through MyPHPAdmin to correct this by hand, but I sure wish it wasn't happening.
Otherwise, I couldn't be more thrilled with CB.