Focal has been pretty open about how they voiced the speakers using beryllium tweeters with the tweeter grills installed. So while they will certainly sound different with the grill off, it wasn't the design intent. Removing the fabric grills from the mids and woofers is good, and generally encouraged.
It's probably a small risk, but remember that Be is a toxic metal, so if you have younger ones in the house that might touch or poke, it's definitely safest to leave the grille on.
Full disclosure: I'm quite happy with the metal tweeter grilles installed on both my Kantas and Sopras.
