# RC4 encryption problem

• Sep 6, 2010, 05:13
nocilis
RC4 encryption problem
Hello,

I'm trying to fix an issue with a classic asp website that uses Mike Shaffer's RC4 algorithm for encryption and decryption:

Code ASP:

```Dim sbox(255) Dim key(255)   Sub RC4Initialize(strPwd) '::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: '::: This routine called by EnDeCrypt function. Initializes the ::: '::: sbox and the key array) ::: ':::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::   dim tempSwap dim a, b, intLength   intLength = len(strPwd) For a = 0 To 255 key(a) = asc(mid(strpwd, (a mod intLength)+1, 1)) sbox(a) = a next   b = 0 For a = 0 To 255 b = (b + sbox(a) + key(a)) Mod 256 tempSwap = sbox(a) sbox(a) = sbox(b) sbox(b) = tempSwap Next   End Sub   Function EnDeCrypt(plaintxt, psw) '::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: '::: This routine does all the work. Call it both to ENcrypt ::: '::: and to DEcrypt your data. ::: ':::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::   dim temp dim a, i, j, k dim cipherby dim cipher   i = 0 j = 0   RC4Initialize psw   For a = 1 To Len(plaintxt) i = (i + 1) Mod 256 j = (j + sbox(i)) Mod 256 temp = sbox(i) sbox(i) = sbox(j) sbox(j) = temp   k = sbox((sbox(i) + sbox(j)) Mod 256)   cipherby = Asc(Mid(plaintxt, a, 1)) Xor k cipher = cipher & Chr(cipherby) Next   EnDeCrypt = cipher   End Function```

I'm basically just passing in a 15 digit card number. Recently, the encryption method seems to fail whilst looping through each character in the EnDeCrypt function. The value returned (cipher) appears to be made up of 15 characters, but some of these are html entities of the encrypted character...not the characters themselves.

It seems to fall down when the value returned by the following line (cipherby) returns 0 (zero):
cipherby = Asc(Mid(plaintxt, a, 1)) Xor k

In other words, the next line, Chr(0), is returning a Null value, which causes all sorts of issues when the returned cipher is fed into an INSERT query.

I have also noticed that the characterset/page encoding affects the encryption of the characters, so I'm just wondering if I'm using the wrong characterset (utf-8) or codebase perhaps...

Any help is appreciated.

Thanks
• Sep 6, 2010, 05:30
siteguru
Looks like this code is intended for single-byte characters sets. So if you use a double-byte character set it will likely throw a wobbly. My guess is that this is expecting a Western Latin-1 type character set.
• Sep 6, 2010, 08:20
nocilis
If I change the charset to ISO-8859-1 it doesn't appear to make much difference to the problem.

Incidentally and for my understanding, can double-byte character sets contain characters in a single-byte representation? i.e. < 256. Because the encryption sometimes does work, which would seem to indicate that in those situations the encryption algorithm is churning out acceptable characters.