If you a casual relaxed driver you won't notice much, but it is the extreme situations that they are talking about, things like emergency maneuvers and accelerating in corners are in serious jeopardy. I am surprised you haven't noticed already. When it hit 45 in NJ, I already knew that something was up when traction control started to kick in, a lot.
You have already made it through half the season. In this weather, you wear the summer compound fast, so just take it very easy or else you will be buying new tires anyway. Personally, I would want to go down to 18" anyway in pothole season. I have hit a couple here in NJ that made me glad I didn't have my stock rims.
My recommendation: Decide at the end of the season if you felt your life was in jeopardy a couple of times, and then buy the winter tires for next season. I put mine on in November.
Also, if you have survived the winter so far with your summer tires, then be sure to just get winter performance tires. Obviously, you don't need a true winter tire, and you will enjoy being able to have summer fun in the winter, safely.