If someone challenges you on the ladder, you can always cancel that challenge (before you make a move), which will result in:
- The game not counting toward any ratings changes (i.e, the kyu-dan system).
- The challenger immediately taking the spot above you on the ladder (if they have not already moved up higher from some other game).
If the second thing does not happen, then it’s probably a bug that should be reported.