ssh - What can a user do without sudo?

I'm setting up a server and want a user to only have access to his own files. I do a regular adduser command:

sudo adduser username

And give the user a password. Then I log in to it via ssh and see that I can cd to / and to other users home folders. I don't seem to have write access to much outside of ~/, but I was just thinking exactly what damage can a new user do without having been added to the sudo group?

Should I be taking more precautions and locking down the user further?

In principle, a user created in that way cannot do any damage outside his/her own home directory. As you noticed, a regular user does have access to various directories (e.g. in /, /usr/, /tmp). This is a necessity, because most user-accessible programs are located in /usr/bin and /bin. If the user didn't have (read-only) access to these directories, he/she wouldn't be able to run any programs.

However, a regular user doesn't have accesss to home directories of other users.

You can use the ls -l command to see permissions on a file or directory. See https://help.ubuntu.com/community/FilePermissions for more information on file permissions.

It is possible to limit an ssh user to only a few programs. See for http://www.pizzashack.org/rssh/ and http://www.cyberciti.biz/tips/linux-unix-restrict-shell-access-with-rssh.html for the rssh tool, which limits the user to only run copying tools like scp and rsync. Such users won't be able to log in normally to get a shell and run other commands.

Another option is to create a 'chroot' or 'jail' environment. See this AskUbuntu question.