Well I mean if you really wanna go down that path, even people who ARE (lets say magically for the moment) good at anything shouldn't bother, because they're gonna die and it won't matter (to them) if they're remembered or not.
The point is, this kind of logic is self defeating in and of itself. The point of doing something isn't really to be remembered, it wouldn't matter to you whether you are remembered or not (even if you believe in an afterlife like heaven, wouldn't the everlasting afterlife become more important to you than the temporary life you lived? If you believe in reincarnation, you don't retain any of the memories, maybe you hear about your past life's achievements, but not knowing that it was actually you just makes it another "famous" person.) The point of doing something is about what enjoyment it creates to you now, while your alive.
You want to feel that you will be remembered because it gives you happiness now, you can quit if you want, but it won't give you happiness in doing so. I have a personal belief that it is better to have tried and failed than not to have tried at all, because even in failure you can create something from it, some experience, some interesting paths, there is no shame in failure of a hobby. If you draw and your bad at it, your still learning something, your still doing a different experience, learning a new skill, you may never be a master, but you would have created something, and even if that something isn't good, its still something, a permanent monument to your journey, a monument not to anyone else, but to you. (And hell, there is a always a chance something you create might actually be interesting to a lot of people!)
Do what makes you happy, if you feel quitting makes you happy, then quit, I'm not going to say you should do things you don't want to do just because you want to be remembered. I'm going to ask you though to really look into yourself and figure out whether quitting is really what you want to do, it might be, but it might be an unrelated solution to a different problem you're having.