apt - Make mesa drivers become independent of previous nvidia installation

I had nvidia driver installed on my Ubuntu Server 16.04. I need to run some app which needs OpenGL, but since machine is "headless" a virtual screen must be created.

I tried to do it with Xvfb, but app fails with ... unable to ... GLX ... error. Internet at ~50 places says it's totally impossible to achieve a virtual screen with nvidia drivers installed.

So I uninstalled drivers with apt remove --purge nvidia*
It didn't fix the problem so I removed nvidia's .so files present at ldd /usr/bin/glxinfo to somehow force Xvfb use mesa's drivers, so now I have:

# ldd /usr/bin/glxinfo
    libGL.so.1 => /usr/lib/libGL.so.1 (0x00007f37f7f50000)

    libX11.so.6 => /usr/lib/x86_64-linux-gnu/libX11.so.6 (0x00007f37f7c16000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f37f784b000)
    libnvidia-tls.so.340.102 => not found
    libnvidia-glcore.so.340.102 => not found
    libXext.so.6 => /usr/lib/x86_64-linux-gnu/libXext.so.6 (0x00007f37f7638000)

now xvfb and glxinfo fail with error while loading shared libraries: libnvidia-tls.so.340.102: cannot open shared object file: No such file or directory error :(

ldconfig doesn't help as well as apt install --reinstall libgl1-mesa-dri xvfb mesa-utils

Is there a way to get rid of remnants of nvidia's drivers? or make xvfb work along with presence of nvidia's OpenCL drivers?