Lily Gardens cost 6 Love are worth 4 Victory Points at the end of the game. In order to purchase a Lily Garden, the number of Gardens and Ririko Hiiragis in your Private Quarters cannot exceed the number of Lily Gardens in your Private Quarters.
If another player uses the ability of Victoria Calderan to destroy a Garden in your Private Quarters, then it is possible that you may end up having more Lily Gardens than you have Gardens and Ririkos. In this event, you do not need to return one of your Lily Gardens to town.
Lily Gardens have one of the highest VP to cost ratios in the game. They'd dominate Expanding the House if that expansion didn't also add a ton of other sources of VP. But yes, you'll want to buy as many of these as possible, and they'll go quickly because of their relatively low price.