Their next few years were up and down