your variable “my” is an associative array. That is it an array where you can access the elements by a string. Your variable “my” is defined a an array with a single element access by “name” with a value the string “mytest”.
You add another element to the array with the statement
my.mytest = function() {
alert(this.name);
}
This element of the array is accessed by “mytest” and has a value of function. That is whenever the element is accessed the function is executed. The value return by the function is the value of that array element.
Your statement var o = new my.mytest(); is causing the function to be executed, which is showing the value of this.name. “this” properly exists as an object but because “name” is not a property of that object you get the “undefined”.