I found I couldn't get colemak to work for me. Like you said, too many key combos that just felt awkward. I wound up using a layout[1] called "middlemak" that feels much more comfortable to me, and preserves a lot of qwerty hand if not actual key positions, which helped in the initial switch. Took me a few months to get up to a relatively normal typing speed and a year in (of maybe 2-4 days of use a week depending on whether I'm working from the road or not) and I still have to look at my keys about 50% of the time, but it's improving and since I only customized this one keyboard, it doesn't seem to have broken my muscle memory for a standard qwerty keyboard.

[1]: https://configure.zsa.io/ergodox-ez-st/layouts/wzKWq/latest/...