Your love relationship is a fantasy and a conditional relationship, and it suits both of you.
Talking to the man about sharing your life on a more permanent basis is a decision only you can make. You can anticipate the outcome of such a conversation if you are quite clear in your own mind about what you want. How do you get such clarity? Find the other side of the fantasy before you discuss this with him.
You are getting many benefits from this relationship, for example the excitement of the secret, discovering your capacity to love, being loved by him, and also the freedom to choose when you want to see him and when you want to be without him, the anticipation of meeting him, and so on.
The downside is that you don't know his family or friends, or anything about his life that he chooses not to share with you. You cannot rely on his support when things go off the rail for you, because that was not part of your ‘contract’ with him. You do not know whether he is a good father. You have always seen him in isolation rather than as part of your complete life.
If you want to discuss with him the possibility of sharing everything in your life, you could scare him off (because you will destroy his fantasy about you) or you could enter a phase of massive adjustment in your life where both of you will have to leave the secrecy and fantasy behind and deal with every wonderful and horrible bit of reality together. That is a massive test for every relationship.
Let's say he also wants to share more with you and become a true part of your life. How would your life change? Do you want all those changes? Some of them will be for the better, while others will be for worse. You might share even more happiness, and you will have to navigate more challenges together. Is that what you want? Because I can guarantee you that no matter how much you love each other, there will be challenges for both of you to overcome. Of course there is also the possibility of even a deeper love.
Now let's say he does not want to change anything, and he does not want to take on the responsibility of your children and another child you might have with him. What if he then decides to end the relationship? This is where you seriously need to look at how you have changed and grown in your relationship with him. Did he help you to understand the real meaning of Love? Did he help you to heal and love yourself? Did he help you to understand and appreciate your freedom of choice?
If he does not share your desire to share your life fully, you may lose him, but you will never lose the wisdom you gained from your relationship with him - and that is the gift from him that you will take into any future relationship.
As long as you are scared of losing him, you are still dependent on him and doubting yourself. Discover what he offers that you are dependent on, and find that in yourself. Then you will no longer be dependent on him, and you will be able to enjoy your time with him being even more yourself, and because you want to rather than because you feel you are incomplete without him.
Whatever you decide and whatever he decides will be right for both of you.
To read more about dealing with relationship issues: