This is one that depends on personal preference, but have you checked out Yoshimo's fighting skills properly? He IS a BLADE! Have you used his Offensive Spin? Or his Defensive Stance? If you don't use these skills, then yes, he is only an average fighter. I prefer him to Jan, myself. And his Identification skills are pretty good, even though less than a regular Bard. I think he is a useful team member.
However, there are a number of other people available to take into your team if you want a change!
F.