Windows 7 CAPS lock

I am a new user to windows 7. Jus noticed a strange icon like shown below. It always stays on top regardless what application I am running.I got curious for a little bit and found out it was the CAPS icon. It should disappear after pressing the CAPS a few times but somehow it didn't happen this time. That is why I got a little curious.