Motorola Droid :: Random Audio Output Switch To Handset

I have a little issue here with my droid's audio output: it seems to be randomly deciding it should come out of the handset rather than the speaker. I notice this happens often when I end a call, that the audio doesn't switch back to the speaker, but it also seems to happen completely at random - for example, it happened at some point when I was asleep last night, which prevented my phone waking me up when my mom was trying to call me this morning, because the ringtone came out of the handset, and thus, was too quiet for me to hear.

Using google voice, I found a workaround - I set voicemails to play through the speaker instead of the handset in the app, so if I go in there and play a voicemail, it'll switch me back to speaker. I can also sometimes get switched back to speaker by making a call, but that's inconsistent, since calls are just as likely to cause the problem as to solve it. I don't know if this is being caused by an app or not.