After a lot of experimentation and research I think I've figured out that I have a lazy left eye. Doctors said I had one when I was a kid and that I would grow out of it. I seemingly grew out of it with no additional treatment.
After more research a lazy eye isn't just an eye that drifts, it's an eye that the brain doesn't want to use. Long story short, binocular vision is tough for me even with both eyes wide open because my brain only wants input from the right eye so I'm VERY right eye dominant.
I can do it but it takes way more effort than most people. As an example I asked my brother in law to look at an occluded red dot for the first time and it immediately worked for him. With practice it's becoming a lot easier to do and I can use an occluded dot with little effort after months of practice. It's definitely helped me be more aware of my vision, if I'm using both of my eyes, and what I'm focusing on.
I plan on getting an eye exam for the first time in probably 20 years and I'll try and remember to update this thread with the results in case anyone else has the same issue as me.