upexec gets error "This program cannot be run in DOS mode"

[hdm at metasploit.com (HD Moore)]

On Tue, 2009-09-08 at 17:10 -0400, Kevin McNamee wrote:
> I am a new Metasploit user and have been experimenting with the upexec
> payload. The target is an XP SP3 system with a vulnerable version of
> netcat installed. I run the ?netcat110_nt? exploit with the payload
> set to ?windows/upexec/reverse_nonx_tcp?. 
[snip]
> 16:30:25 ? netcat110_nt [*] Executing uploaded file?    
>
> 16:30:25 ? [*] Session 1 created for 192.168.0.2:1038    

> Any advice on how to fix this or on what debugging steps to take next
> would be greatly appreciated.

Try to locate the temporary file that was uploaded to the disk and
examine it - it sounds like either the EXE is broken on your system, the
upload process is corrupting it, or something on the remote end is
mangling it once uploaded (antivirus, etc). You would be better off
using the Meterpreter payload and manually uploading/executing a
payload, or using the upexec Meterpreter script. Both of these should
make the diagnostic process a bit easier, and be more forgiving in the
case of errors.

-HD