Pruning woody branches/stem is not a problem. If you are taking out a branch, cut it almost flush an leave a tiny stub. Kaffir has a weeping tendency as you can see from your plant. They tend to produce long branches that eventually droop and as they harden (become woody) that shape becomes more permanent.
In this case, pruning is used to "chase back" the branch. That is you would prune back to a few (1 to 3) leaf nodes on the branches. This will encourage more branching and tighter plant.
Keep the lower branches . I would not prune it completely. They help to strengthen the main trunk. But it is your choice. Some like a straight bare trunk with foliage near the top. But it will make the stem weaker and increase tendency to bend without any support.
For top I would chase it back hard to a point where there is one branch sticking upwards. That will become the new leader. A leader is the branch that grows straight up and become the main trunk line. Imagine a big tree - they are always wider near the base with thicker branches . As you go near the top branches are thinner and it gets narrower near the tip. For woody plants this is a stable configuration. If you leave many branches near the top then they will be slow to harden. That is you select a leader so that it grows faster and hardens quicker.
You can try straightening the branches near the top also. Choose you would like to straighten up that has started to harden. Stick a long strong pole on the opposite side in the pot so that it angles outward following the angled sides of the pot. Stick it it all the way to the bottom so it it is less likely to move. Then tie the branch to the pole pulling to a straighter angle. You may need two tie points. If you feel the branch will break then do it in stages. Then let it grow and harden. And pull it backwards a bit every few weeks.
Q