To my mind, .XYZ has a disposable, throwaway quality to it. Those connotations are not irrational, actually. Since the .XYZ registry flooded the internet with innumerable .XYZ domains at 1 penny apiece, that extension has a real association with spammers. Also, it's a bit unwieldy in conversation. Saying "dot ex why zee" adds 3 syllables to the brand name without contributing any meaningful idea.
If you decide to upgrade or rebrand, I'd be happy to point out a wider range of options. .COM is, of course, the global default. Co-founders, sponsors, interviewees, and readers would probably take your project more seriously if it were built on that stable footing. But you don't necessarily need to display .COM. For a fashion website, plenty of other domain endings deserve consideration too: .STYLE, .FASHION, .DESIGN, .CLOTHING, .ME. Any 1 of those might be a better fit than .XYZ.
There are some pitfalls in this area, though. Choosing a domain / brand name is definitely worth a conversation.