validation of user input: Too Much validation? Pls help..

[sprockett]

Hi I have 2 qns....

I'm writing the api for db read/write and im confused abt a couple of things..

1. Where should validation of input occur?

I'm writing functions to check every single incoming field: userid, date, password, etc. I was thinking if i check user input the moment he calls a "write to db" function, would i then need to do more validation on the html form itself? if i do so, wouldnt i be checking twice, which is wasting proc. time?


2. Currently, apart from field specific data validation of format(which im using REGEX for), I'm also validating for correct length and correct type (strlen()/is_string()/is_int()), along with the whole ADDSLASHES, urlencode shebang.
Am I putting in too much validation? I fear by the time I'm done, it will take a user 10 seconds to add some data, coz of all these checks..


would appreciate help!
thanks!

[Daimaju]

I think the db api/object should only make sure that the data is properly escaped, the check for correct length, type, format, etc should be done wiht another (Validator?) class. Search sitepoint for "form validation" and you should get a lot of ideas on who such a class could/should look like.

hth

[sprockett]

Thanks Daimaju,

Problem is i intend to release the api to external parties to play around with too..

so that would mean everything would have to be self contained in the API. validation and db queries all packaged in one.

i was thinking of having a separate validation class too, that acts as an intermediary btw form, and api. However, because of the complex way my table relations are set up, its not that easy.

For example, its not just format/data type(length) validation.
I have some tables that are related to each other (m:n) so there is a intermediate mapping table to allow for this. So i can't have the user map something to an ID that doesnt exist.

So if you realize, its not just a simple "Take data from forms, input into table" idea. There's a lot of back and forth going on... Data validation, checking if records exist etc.

Things would be a lot simpler if i wasnt going to release the api, then the api wouldnt have to check everything.

Sigh.. Would appreciate any more help/comments!

Thanks guys, i really don't know what i'd do w/o sitepoint.

I think the db api/object should only make sure that the data is properly escaped, the check for correct length, type, format, etc should be done wiht another (Validator?) class. Search sitepoint for "form validation" and you should get a lot of ideas on who such a class could/should look like.

hth

[bigduke]

Having an API does not mean you limit the number of classes. If you really wish to do a good OOP design then give each class the functionality it desrves, no more and no less. The entire API can be a collection of function calls from various classes like in this case the validator class and the db class.
Also, elaborating on your "complex" validation requirements, subclass relationValidator class from validator and carry out the operations anyway.

[McGruff]

User input crosses a boundary between the outside world and the internal php application. I'd suggest validating input in some kind of request model so that nothing bad is allowed in. Inside the app, test everything thoroughly to make sure "bad" data is never created by the app itself. When you come to update the db, you won't need to validate.

[sprockett]

Many many thanks guys for the extremely helpful replies... Here's a simple example of my db
User Table
-UserID
-UserName

CAR Table
-CarID
-Name
-TypeID

UserCarMap Table
-UserCarMapID
-CarID
-UserID
-description

TYPE Table
-TypeID
-Name

TAG Table {User can assign TAGS to the car, so he can easily search via tags later}
-TagID
-TagString

UserCar-TagMap Table
-UserCar-TagMapID
-UserCarMapID
-TagID


Ok, so the form basically allows the user to enter a new car.
- validation for all the main string fields are easy as pie... no problem.
- For the TYPEID field, he is presented with a combo box with available "TYPES" (but i will STILL conduct a relationValidation, coz i can't just rely on the fact that the html form has the right values in it...)

-Now come's the "TAG" field. The form will allow him to enter a string in the folllowing form:
nice BMW wishlist "very nice",
and it will store the following tags:
1. nice
2. BMW
3. wishlist
4. very nice


So bascially when a user submits a new car, the following should happen:


1. Check if specified USERID to add car for exists. Return error if it doesnt
2. Check if Car with carname specified by user exists in CAR TABLE {
return CARID of existing car
} else {
create Car with userspecified carname;
return CARID of newly created car;
}

2. Check to see if user has already added car {
Query User-CarMap table to ensure carID is not mapped to UserID
}


3. Proceed to TAG the User-Car Mapping with array of TAGS {
For each of tags in tag array {
if TAG has already been created {
return existing tagid;
} else { //tag already exists!!
create new Tag;
return Newly created tagid;
}
check if UserCar-TagMapping already exists
if not, insert UserCar-TagMap table with new mapping
}



This is a simple example of why im getting confused about where validation occurs...
My DB actually has a few more orders of relational complexity (argghh)

Don't get me wrong, im not saying that EVERYTHING Is hardcoded into my API (with no function calls)
I actually have separate little validation/adddata functions which are called from the API.

But Im packing EVERYTHING into the API, so someone can just send in a CAR ID, USER ID, and an array of tags. And the api will settle everything... from checking of relations, data type/formatting checking etc.

But im not sure if what is the best way to go, because i feel by making my API do everything, there may excessive redundant calls to the validation/checking functions, especially if im gonna include javascript validation in teh form itself...

Im so confused im tearing my hair out!!

[McGruff]

Is this all in a single class?

[sprockett]

i have a class for my data object.

but no classes for anything else yet... i've written out lots of functions though. Still figuring out the optimal way to group what functions into what classes. eg, class that deals with DB, class the validates etc.

Regarding my "Data object" class, an example would be a
class UserCarMapping class.

Each "User-CarMapping" object will have the following member variables:
1. UserID
2. UserName
3. CarID
4. CarName
5. UserCarMapping ID
6. UserCar Map description
7. Array of Tags that user has given to this particular UserCarMapping.

Data Input:
Basically, the form will populate an array of these objects. THis array of objects then get passed to the ADDOBJECTTOdatabase function, which will then deal with adding all the separate info into the relevant tables, and ensure mapping is ok.

Data Display:
Similar to data retrieval. Depending on what the user wants to display, a request is sent to the "retrieveobjectdata" function, which then populate an array of the abovmentioned object, and the form will then deal with displaying data from the array of objects.

I think im in way over my head with this project...

Is this all in a single class?

[mlemos]

1. Where should validation of input occur?

I'm writing functions to check every single incoming field: userid, date, password, etc. I was thinking if i check user input the moment he calls a "write to db" function, would i then need to do more validation on the html form itself? if i do so, wouldnt i be checking twice, which is wasting proc. time?

Validation should always happen on the server side in PHP before the data is processed and submitted to the database.

However, whenever possible you should also validated on the client side using Javascript to make your forms more usable by avoiding the server submission round trip just to tell the users that there are invalid fields.

You may want to take a look at this forms generation and validation class that can simplify your task by taking a few validation definitions that you provide to validate forms on the server side in PHP with the class itself and also on the client side using Javascript generated automatically by the class as part of the form output.

2. Currently, apart from field specific data validation of format(which im using REGEX for), I'm also validating for correct length and correct type (strlen()/is_string()/is_int()), along with the whole ADDSLASHES, urlencode shebang.
Am I putting in too much validation? I fear by the time I'm done, it will take a user 10 seconds to add some data, coz of all these checks..

You should never accept data that is invalid with the fear it may take too much time to validate, or else you may open security holes in your site.

Still, the kind of validations that you mentioned are not heavy. What is heavier is to check the database or some remote service to verify your data. Still that should be done as I mentioned above.

[sprockett]

Hi Manuel,

One trillion thanks for your reply. It clarifies many things. I feel im on the righth path making sure everything is valid right now.

Thanks for the link too! Will definitely be using that for my forms.

Validation should always happen on the server side in PHP before the data is processed and submitted to the database.

However, whenever possible you should also validated on the client side using Javascript to make your forms more usable by avoiding the server submission round trip just to tell the users that there are invalid fields.

You may want to take a look at this forms generation and validation class that can simplify your task by taking a few validation definitions that you provide to validate forms on the server side in PHP with the class itself and also on the client side using Javascript generated automatically by the class as part of the form output.


You should never accept data that is invalid with the fear it may take too much time to validate, or else you may open security holes in your site.

Still, the kind of validations that you mentioned are not heavy. What is heavier is to check the database or some remote service to verify your data. Still that should be done as I mentioned above.

[Dr Livingston]

1. Where should validation of input occur?

I'm writing functions to check every single incoming field: userid, date, password, etc. I was thinking if i check user input the moment he calls a "write to db" function, would i then need to do more validation on the html form itself? if i do so, wouldnt i be checking twice, which is wasting proc. time?

I have a Request object as suggested to sanitise the data first of all, and thereafter I validate the data based on the number of characters inputted for example, ie A Size Rule?

Now, I have an XML file something like this,

<forms>
	
	<form name="Contact">
		<input name="Forename" required="1">
			<filters>
				<filter name="sizeRange" minimum="6" maximum="32">
					<error>You need to enter a minimum of 6 characters and a maximum of 32 characters.</error>
				</filter>
				<filter name="regExpression" regexp="/[a-zA-Z]+$/">
					<error>You have entered an illegal character(s).</error>
				</filter>
			</filters>
		</input>
		<input name="Surname" required="1">
			<filters>
				<filter name="sizeRange" minimum="6" maximum="32">
					<error>You need to enter a minimum of 6 characters and a maximum of 32 characters.</error>
				</filter>
				<filter name="regExpression" regexp="/[a-zA-Z]+$/">
					<error>You have entered an illegal character(s).</error>
				</filter>
			</filters>
		</input>
...

Whereby I then have an XmlWalker to walk the XML file, and an Adapter to take a certain XML Node, and parse it. The Adapter has a Visitor, which is visited once the XML Node has been parsed.

It's this Visitor that does the validation by loading a class for each filter required given for each form input. The validated inputs, errors, etc etc reside in this Visitor alone, so it's a case of pulling out the validated data at the end of the day, ready to be inserted or updated to the database.

Hope this helps, and gives you some ideas

EDIT:

When I talk about an Adapter, I mean this is the name of the class and not the name of the Design Pattern, which is not related, so no need to read up on this design pattern.

I can't post any script though due to contractual reasons, I can only give some clues and maybe some help if you need it

[DarkAngelBGE]

Maybe you would like to use this class:

<?php
/**
 * Author: Tim Koschützki
 * Based on ideas from Matt Mecham
 * Date Started: January 7th, 2003
*/
class Request {

    /*-------------------------------------------------------------------------*/
    // Makes incoming info "safe"              
    /*-------------------------------------------------------------------------*/
    
    public function parse() {        
        $this->get_magic_quotes = get_magic_quotes_gpc();
        
        $return = array();
        
        if( is_array($_GET) ) {
            while( list($k, $v) = each($_GET) )    {
                if ( is_array($_GET[$k]) ) {
                    while( list($k2, $v2) = each($_GET[$k]) ) {
                        $return[ $this->clean_key($k) ][ $this->clean_key($k2) ] = $this->clean_value($v2);
                    }
                } else {
                    $return[ $this->clean_key($k) ] = $this->clean_value($v);
                }
            }
        }
        
        //-----------------------------------------
        // Overwrite GET data with post data
        //-----------------------------------------
        
        if( is_array($_POST) ) {
            while( list($k, $v) = each($_POST) ) {
                if ( is_array($_POST[$k]) )    {
                    while( list($k2, $v2) = each($_POST[$k]) )    {
                        $return[ $this->clean_key($k) ][ $this->clean_key($k2) ] = $this->clean_value($v2);
                    }
                } else {
                    $return[ $this->clean_key($k) ] = $this->clean_value($v);
                }
            }
        }
        
        $return['request_method'] = strtolower($_SERVER['REQUEST_METHOD']);
        
        return $return;
    }
    

    private function clean_key($key) {
        if ($key == "") {
            return "";
        }
        
        $key = htmlspecialchars(urldecode($key));
        $key = preg_replace( "/\.\./"           , ""  , $key );
        $key = preg_replace( "/\_\_(.+?)\_\_/"  , ""  , $key );
        $key = preg_replace( "/^([\w\.\-\_]+)$/", "$1", $key );
        
        return $key;
    }
    
    private function clean_evil_tags( $t ) {
        $t = preg_replace( "/javascript/i" , "javascript", $t );
        $t = preg_replace( "/alert/i"      , "alert"          , $t );
        $t = preg_replace( "/about:/i"     , "about:"         , $t );
        $t = preg_replace( "/onmouseover/i", "onmouseover"    , $t );
        $t = preg_replace( "/onclick/i"    , "onclick"        , $t );
        $t = preg_replace( "/onload/i"     , "onload"         , $t );
        $t = preg_replace( "/onsubmit/i"   , "onsubmit"       , $t );
        $t = preg_replace( "/<body/i"      , "&lt;body"            , $t );
        $t = preg_replace( "/<html/i"      , "&lt;html"            , $t );
        $t = preg_replace( "/document\./i" , "document."      , $t );
        
        return $t;
    }
    
    private function clean_value($val) {       
        if ($val == "")    {
            return "";
        }
    
        $val = str_replace( " ", " ", $val );
        $val = str_replace( chr(0xCA), "", $val );  //Remove sneaky spaces
        
        $val = str_replace( "&"            , "&amp;"         , $val );
        $val = str_replace( "<!--"         , "<!--"  , $val );
        $val = str_replace( "-->"          , "-->"       , $val );
        $val = preg_replace( "/<script/i"  , "<script"   , $val );
        $val = str_replace( ">"            , "&gt;"          , $val );
        $val = str_replace( "<"            , "&lt;"          , $val );
        $val = str_replace( "\""           , "&quot;"        , $val );
        $val = preg_replace( "/\n/"        , "<br />"        , $val ); // Convert literal newlines
        $val = preg_replace( "/\\\$/"      , "$"        , $val );
        $val = preg_replace( "/\r/"        , ""              , $val ); // Remove literal carriage returns
        $val = str_replace( "!"            , "!"         , $val );
        $val = str_replace( "'"            , "'"         , $val ); // IMPORTANT: It helps to increase sql query safety.
        
        // Ensure unicode chars are OK
        
        if ( $this->allow_unicode ) {
            $val = preg_replace("/&amp;#([0-9]+);/s", "&#\\1;", $val );
        }
        
        // Strip slashes if not already done so.
        
        if ( $this->get_magic_quotes ) {
            $val = stripslashes($val);
        }
        
        // Swop user inputted backslashes
        
        $val = preg_replace( "/\\\(?!&amp;#|\?#)/", "\", $val ); 
        
        return $val;
    }
}

Simply use it with:

// parse input data
require_once(LIB_PATH.'class.Request.php');
$request =& new Request();
$INPUT = $request->parse(); 

[sprockett]

many thanks all! all of u are life savers...