I want to change my status from H4 to F1 and assuming that it would take approx 3-4 months. If I apply for COS today, I make a trip to France (I am French citizen) next week, can I re-enter US on my valid H4 visa? Or do I have to wait until F1 is approved, get it stamped in Paris and then re-enter US?
You do NOT need USCIS approval to apply for F-1 visa. Just apply for F-1 visa during your visit to France. Ask the school to explain the formalities to you. By the way, if you leave USA during a COS request, the COS is deemed abandoned.