The pandemic pushed millions of older Americans out of the labor force and many retired early.