It does happen to me too, but on my iPhone. When I leave my office desk with my phone and get back, the connection is lost and the only way to link back is as described by @Mozart above.
I believe this has something to do with an iOS update as the thing worked like a charm for months until May...