Hello,

I'm trying to fix an issue with a classic asp website that uses Mike Shaffer's RC4 algorithm for encryption and decryption:

Code ASP:
Dim sbox(255)
   Dim key(255)
 
   Sub RC4Initialize(strPwd)
   ':::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
   ':::  This routine called by EnDeCrypt function. Initializes the :::
   ':::  sbox and the key array)                                    :::
   ':::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
 
      dim tempSwap
      dim a, b, intLength
 
      intLength = len(strPwd)
      For a = 0 To 255
         key(a) = asc(mid(strpwd, (a mod intLength)+1, 1))
         sbox(a) = a
      next
 
      b = 0
      For a = 0 To 255
         b = (b + sbox(a) + key(a)) Mod 256
         tempSwap = sbox(a)
         sbox(a) = sbox(b)
         sbox(b) = tempSwap
      Next
 
   End Sub
 
   Function EnDeCrypt(plaintxt, psw)
   ':::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
   ':::  This routine does all the work. Call it both to ENcrypt    :::
   ':::  and to DEcrypt your data.                                  :::
   ':::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
 
      dim temp
      dim a, i, j, k
      dim cipherby
      dim cipher
 
      i = 0
      j = 0
 
      RC4Initialize psw
 
      For a = 1 To Len(plaintxt)
         i = (i + 1) Mod 256
         j = (j + sbox(i)) Mod 256
         temp = sbox(i)
         sbox(i) = sbox(j)
         sbox(j) = temp
 
         k = sbox((sbox(i) + sbox(j)) Mod 256)
 
         cipherby = Asc(Mid(plaintxt, a, 1)) Xor k
         cipher = cipher & Chr(cipherby)
      Next
 
      EnDeCrypt = cipher
 
   End Function

I'm basically just passing in a 15 digit card number. Recently, the encryption method seems to fail whilst looping through each character in the EnDeCrypt function. The value returned (cipher) appears to be made up of 15 characters, but some of these are html entities of the encrypted character...not the characters themselves.

It seems to fall down when the value returned by the following line (cipherby) returns 0 (zero):
cipherby = Asc(Mid(plaintxt, a, 1)) Xor k

In other words, the next line, Chr(0), is returning a Null value, which causes all sorts of issues when the returned cipher is fed into an INSERT query.

I have also noticed that the characterset/page encoding affects the encryption of the characters, so I'm just wondering if I'm using the wrong characterset (utf-8) or codebase perhaps...

Any help is appreciated.

Thanks