The pellets must be loaded a particular way. One end is designed to be easily ignited. Could it be you were loading them backwards?
There is no need to remove the cylinder to load it. The old timers certainly didn't (with the exception of the 1849 Colts who had no rammer).
Been shooting cap and ball revolvers for 40-plus years. I rarely remove the cylinder to load.
This affectation with separate loaders puzzles me. I guess they're useful if you compete and need to load a number of cylinders, but in most instances you can load just fine with the rammer built on the gun.
I don't often shoot conical bullets, which can be overly long and resist fitting under the rammer. I shoot balls almost exclusively: .454 or .457 inch for my .44s, and .380 inch for my .36 calibers.
The slightly oversized ball ensures a good, tight fit in the chamber. I also believe that being oversized, the rammed ball creates a wider bearing band for the rifling to grasp. It also seals the ball better in the bore (obturation).
Pellets are far more expensive than loose powder. Get a good powder flask, fill it with FFFG black powder or Pyrodex P, and use a spout of the proper capacity.
You'll find this too is quick and easy.
A folding, wooden pistol holder that holds the pistol pointed upright makes loading much easier than trying to hold it in your hand.
If you use a felt wad on top of the powder, load it as a separate operation. You'll get a better feel for how much pressure you're applying to the wad and powder beneath, and the first time you find yourself ramming a wad down an empty chamber, you'll thank me. It's much easier to remove a felt wad from the chamber than a ball.
After seating the wads in the charged chambers, then seat the ball. Here again, you'll have a better feel for how much pressure you're applying to the ball.
The wad should be seated on the powder firmly, to the point of a slight crunch. The ball should be seated firmly on the wad, but not rammed down hard. Hard ramming can crush the powder granules and affect the burning rate.
Consistency is key to accuracy.
You'll find all kinds of suggestions, claims, rumors, myths, speculation, tips, historical inaccuracies, darned good advice and outright lies at the shooting range, in books and on the internet -- all dealing with cap and ball revolvers.
Some will be apparent, others will require research or the application of common sense.
You've entered a fascinating hobby, my friend.
Hemingway observed that the singlemost attribute for a good writer is a, "built-in, shockproof, s**t detector."
It'd be a mighty handy thing for cap and ball shooters to have too!