320 is pretty coarse unless there are some really deep pits in the barrel. I would typically start a polishing project like that with 400, then step up to 600, and finish up with 1000/1200 or 1500 grit.
I have good luck with just plain old wet-or-dry sandpaper used for automotive refinishing. And yes I use water to lube the sandpaper as I'm working...I just make sure it gets completely dried off when I'm finished up for the day. Use a hairdryer or heat gun to dry it.
That is a nice looking rifle except for the barrel (which doesn't look that bad from your pic). I've salvaged worse!
Any idea when it was built? I'm assuming it was pre-'69 (pre-.25-06Rem) since you're calling it a .25 Neider.
I love the .25-06 nearly as much as I love all of my .264" bore cartridges. I'm partial to almost all of the medium-bores though.
And yup, a range report is definitely needed once you get her all cleaned up! Aw heck, shoot it first...clean it up later!