But, what you're describing isn't really polyamory. An open marriage, yes; ethical non-monogamy, yes. What makes your situation so freeing and uncomplicated is that you don't seem to have any deep emotional commitment to your extra-marital relationship(s).
Polyamory, on the other hand, includes some level of emotional attachment and commitment, certainly an agreement and/or understanding among the partners involved in the polyamorous relationship. After all, the love element is right there in the root of the word itself: amor.
And, when emotions (other than pure pleasure and joy and passion) come into play, things invariably become more complex and more challenging to negotiate.
Another aspect you don't even mention is the risk of STIs. There are some very virulent sexually transmittable germs out there. And, I do think taking precautions and getting tested is worth mentioning. There is no such thing as safe sex. Safer sex, yes. But, there is always a risk. A buzz kill, I know. But, still...
Regardless, I salute your willingness to abandon societal expectations of what a marriage should be and for carving your own path. Your lived example is a very positive one. Enjoy!