Finally found the missing step that I needed to do in order to be able to connect via VNC through the ethernet cable, and be independent of wifi!!
sudo vi /boot/cmdline.txt
ip=169.254.198.106 to the end of the line, all on one line.
hardcode an IP here to connect via Ethernet. Then RealVNCViewer will find it. Woohoo!