You need two pointers working at different speeds. A read pointer scans through the array to identify groups of identical characters. A write pointer trails behind, overwriting the array with the compressed output.
For each group, the write pointer places the character first. If the group has more than character, it then writes each digit of the count. Because the compressed form is never longer than the original, the write pointer never overtakes the read pointer.
When the read pointer finishes, the write pointer's position tells you the new length. That's your return value.