Currently the only way to secure the term store is by Term Group. It would be fabulous if that could change to accommodate security at the Term Set level. The reason for this is simple: we have a large site collection with lots of sites. Each of those sites has a different group of contributors. If we open up the site collection Term Group to one group of contributors, they can see and edit all the other sites' terms because they're all accessible within the site collection. However, we already have term sets that pertain only to specific sites, so being able to make each of those editable only by the proper group would be very beneficial.

If I remember correctly, we also can't refer to SharePoint groups in term store security settings. Having this ability would also greatly simplify access controls.