command line - Grep exclude wildcards not working

I am using Ubuntu 14.04 LTS.

I am playing around with grep and I hit an issue piping grep to ls.
All examples are limited to a current directory, non-recursive.

If I want to list all files containing a digit in their name:

ls | grep [0-9]

If I want to list all files containing "d", "h", "m" as a first letter:

ls | grep ^[dhm]

If I want to list all files containing a digit as their first letter:

ls | grep ^[0-9]

However, when I try to use ! inside [], the ! is not included:

user@host:~$ ls | grep [0-9]
1.log
2.log
3.log

hs_err_pid4015.log
Untitled 1.odt
v1.odt

user@host:~$ ls | grep [!0-9]
1.log
2.log
3.log
hs_err_pid4015.log
Untitled 1.odt

v1.odt

The same implies if I try to use it with ^

user@host:~$ ls | grep ^[0-9]
1.log
2.log
3.log


user@host:~$ ls | grep ^[!0-9]
1.log
2.log
3.log

I have spent several days reading other posts, man pages, articles etc regarding grep and wildcards, but I can't figure it out.
I have tried putting them in ' and ", combining both wildcards flags (^ and !) in [[]] etc.

Nothing shows different output that the examples above.

Please, don't show me the solution.
Explain my mistakes and let me figure it out on my own.