
Exploring SAS Syntax and Methods Related to Automatic 
Model Selection
======================================= 11/24/08

Recall that:

   Rsq = 1 - RSS/CSS

   AIC = n*log(RSS/n) + 2p

   Cp = RSS/sighat**2 -n + 2p


/* The following simulated model is like the one in MultReg3.Log
  except that we include a dummy variable for the integer Factor
  variable Lab with levels = 1,2,3.

libname home "SASdat";

data home.simfilD (drop=seed i INT);
    seed=57777;
    do i=1 to 1200; x = ranuni(seed); end; /* burn-in */
    do i=1 to 600;
       x = -0.5*log(1- ranuni(seed));
       if ranuni(seed) < 0.4 then w = 1;
       else w=0;
       Lab = int(ranuni(seed)*3)+1;
       if Lab=1 then INT=3;
       else do; if Lab=2 then INT=1.8; else INT=3.8; end;
       y = INT+ 2*X - X**2 + 1.5*(X**2)*W + 1.2*rannor(seed);
       xsq = X**2; XW = X*W; XSQW = (X**2)*W; XCU = X**3;
       XCUW = XCU*W;
    output; end;      /* produces 600 obs, 9 variables */


  data worksim;
      set simfilD;
      if Lab=2 then do; D2=1; D3=0; end;
         else if Lab=3 then do; D3=1; D2=0; end;
              else do; D2=0; D3=0; end;
   /* The previous 3 lines could also be replaced by:
        select;
           Lab=1; do; D2=0; D3=0; end;
           Lab=2; do; D2=1; D3=0; end;
           Lab=3; do; D2=0; D3=1; end; end;  
   */  run;

  proc glm data=worksim;
    class Lab;
    model y = x Lab Xsq W XW XsqW Xcu / solution; 
        run;


The GLM Procedure

Dependent Variable: y
                                        Sum of
Source                      DF         Squares     Mean Square    F Value    Pr > F

Model                        8      905.926843      113.240855      86.66    <.0001
Error                      591      772.275726        1.306727
Corrected Total            599     1678.202570


R-Square     Coeff Var      Root MSE        y Mean
0.539820      32.43347      1.143122      3.524512


Source                      DF       Type I SS     Mean Square    F Value    Pr > F
x                            1     178.2064946     178.2064946     136.38    <.0001
Lab                          2     442.9730791     221.4865396     169.50    <.0001
xsq                          1      18.3367332      18.3367332      14.03    0.0002
w                            1      44.3436720      44.3436720      33.93    <.0001
XW                           1     198.3844326     198.3844326     151.82    <.0001
XSQW                         1      23.1198572      23.1198572      17.69    <.0001
XCU                          1       0.5625744       0.5625744       0.43    0.5120


/* NOTE that the type I SSQ adds in each variable one at a time, so
     the incremental DF value is always 1 except for a CLASS variable
     like LAB.  
   There is syntax in PROC REG but not in PROC GLM with model-selection for 
   entering variables in all-or-none blocks with { }. */


The GLM Procedure
Dependent Variable: y

Source                      DF     Type III SS     Mean Square    F Value    Pr > F

x                            1       7.4450736       7.4450736       5.70    0.0173
Lab                          2     371.7188607     185.8594303     142.23    <.0001
xsq                          1       0.6703561       0.6703561       0.51    0.4741
w                            1       1.9404823       1.9404823       1.48    0.2235
XW                           1       1.3405922       1.3405922       1.03    0.3115
XSQW                         1      19.3863385      19.3863385      14.84    0.0001
XCU                          1       0.5625744       0.5625744       0.43    0.5120

                                      Standard
Parameter           Estimate             Error    t Value    Pr > |t|

Intercept        3.806403460 B      0.14572053      26.12      <.0001
x                1.401838319        0.58729443       2.39      0.0173
Lab       1     -0.727017645 B      0.11556974      -6.29      <.0001
Lab       2     -1.911319444 B      0.11445862     -16.70      <.0001
Lab       3      0.000000000 B       .                .         .
xsq             -0.466055575        0.65069488      -0.72      0.4741
w               -0.213672424        0.17534205      -1.22      0.2235
XW               0.551266812        0.54425942       1.01      0.3115
XSQW             1.131412701        0.29374170       3.85      0.0001
XCU             -0.119798559        0.18258035      -0.66      0.5120

/* NOTE: the F-values  in the Type III SSQ are actually the squares of
       the t-values in all of the 1 DF model terms, so the corresponding
       non-intercept p-values are identical in the two displays.    */

* ###======== Next comes model selection ==============#### ;


proc reg data=worksim;
    model y = {x xsq} {D2 D3} w xcu xw xsqw  / 
              selection = forward slentry=.02;
   run;
                             Analysis of Variance

                                    Sum of           Mean
Source                   DF        Squares         Square    F Value    Pr > F

Model                     5      903.64404      180.72881     138.60    <.0001
Error                   594      774.55853        1.30397
Corrected Total         599     1678.20257

                        Parameter     Standard
Variable                 Estimate        Error   Type II SS  F Value  Pr > F

Intercept                 2.95757      0.10865    966.19809   740.97  <.0001
--- Group GROUP1 ---                               74.90993    28.72  <.0001
x                         1.90827      0.25319     74.07243    56.81  <.0001
xsq                      -0.94971      0.13589     63.68989    48.84  <.0001
--- Group GROUP2 ---                              376.74891   144.46  <.0001
D2                       -1.19177      0.11470    140.76905   107.95  <.0001
D3                        0.72358      0.11528     51.37428    39.40  <.0001
--- Group GROUP6 ---                              264.12773   202.56  <.0001
XSQW                      1.38343      0.09720    264.12773   202.56  <.0001


                           Summary of Forward Selection

        Group      Number     Partial      Model
Step    Entered    Vars In    R-Square    R-Square     C(p)      F Value    Pr > F

  1     GROUP6         1       0.2631      0.2631     350.369     213.52    <.0001
  2     GROUP2         3       0.2307      0.4938     58.0733     135.83    <.0001
  3     GROUP1         5       0.0446      0.5385      4.7470      28.72    <.0001


*########=============================############# 
We can see here that the forward selection picks XSQW but not W or XW,
and that is probably not something we would allow, because of the
difficulty of interpreting the result.                            ;

proc reg data=worksim;
    model y = x xsq D2 D3 w xcu xw xsqw  / 
              selection = CP  include=4;
   run;


NOTE: The variables in the 4 variable model are included in all models.

Number in
  Model          C(p)    R-Square    Variables in Model

       4     204.8762      0.3811    x xsq D2 D3
       5       4.7470      0.5385    XSQW
       6       6.2364      0.5389    w XSQW
       6       6.4913      0.5387    XCU XSQW
       6       6.7468      0.5385    XW XSQW
       7       7.4305      0.5395    w XW XSQW
       7       8.0259      0.5390    w XCU XSQW
       7       8.4850      0.5387    XCU XW XSQW
       8       9.0000      0.5398    w XCU XW XSQW
       7      21.8358      0.5283    w XCU XW
       6      23.1235      0.5257    w XW
       6      41.2875      0.5116    XCU XW
       5      43.9145      0.5080    XW
       6     161.6439      0.4178    w XCU
       5     172.9413      0.4075    w
       5     191.2864      0.3932    XCU

* The XsqW model is still best, and that would happen also with
  backward selection !  ;


proc reg data=worksim;
    model y = x xsq D2 D3 w xcu xw xsqw  / 
              selection = backward  slstay=.01;
   run;

                          Summary of Backward Elimination

        Variable     Number     Partial      Model
Step    Removed      Vars In    R-Square    R-Square     C(p)      F Value    Pr > F

  1     XCU              7       0.0003      0.5395      7.4305       0.43    0.5120
  2     XW               6       0.0006      0.5389      6.2364       0.81    0.3695
  3     w                5       0.0004      0.5385      4.7470       0.51    0.4749


