I had an odd issue just now when using Gvim. Usually, backspacing when in insert mode (i:set backspace=indent,eol,start
To fix this permanently, add
set backspace=indent,eol,start
to GVim's _vimrc file (which is usually located at C:\Program Files (x86)\Vim\_vi).
mrc
You might want to do this when running the editor as an administrator.