So my sister's boyfriend is over today, and right now they're playing cards or something in the next room.

What I'm sort of uncomfortable about is that earlier, he was trying to find out what this picture of her she mentioned that had been taken at camp(to a female friend of hers, who was also over at that time) was. She didn't want to tell him. I was sort of curious also. It never got to the point of direct confrontation; it mainained a sort of joking atmosphere all the way through, but after asking her for a while, he was sort of kneeling on her requesting a description of the picture.

There didn't seem to be any anger or serious emotion on either side, and my sister didn't seem to be actively trying to get away or anything, and was laughing through most of it, so I didn't do anything, but at one point she asked me "a little help here?" seeming sort of serious, but not quite upset. She didn't seem angry or scared or anything. I didn't know what to do (If it had appeared entirely serious, I would have punched the guy, but it didn't).

So I pointed a flashlight in both of their eyes and handed it to her. About a minute later, I mumbled some passive-aggressive thing like "maybe you shouldn't...". I really didn't know what to do. At no point did it seem like they were actually fighting, or expressing serious emotions, but it also seemed that my sister really didn't want to tell him what she was doing in the picture in question, and that the guy wasn't going to let her up until she did (which is what happened, in the end). The picture was of her licking chocolate out of her nose, apparently.

I really wasn't comfortable with the situation, but niether did it seem like it would have been appropriate to physically intervene. Afterwards, my sister asked me "why didn't you help?" and I answered that I wasn't sure whether they were being serious or not. I really don't understand people when they don't appear to be speaking seriously, and I didn't want to act in a state of confusion, especially when action would mean physical aggression.

Should I have done something? SHOULD I do something? It seemed like they were just joking or something (she didn't seem upset with him at all), but in the end, what happened was that he used force to make her tell him something. On the other hand, no one seemed to think of it as all that big a deal. I'm thinking that it might be none of my business, and that really, there's nothing I could do even if I think I should.

I don't know, I'm confused. Should I just forget about it and leave the situation alone?

It's not your place to be her safeguard, but just ignoring her would have been worse. you acted as you saw fit, which was the most you could have done.

Just tell her that next time, if she makes it more clear that she's serious, you'll be more then happy to step in, but if she's being unclear about it, you're not going to intervene. don't let yourself become a scapegoat here.