Use neuronetwork(tensorflow) for regression 2

Alright, I also try to see what if only features (x) are normalized, y range in previous formula is too close to 1. So I change the formula to be

df['y'] = 10*3*df['x0']+10*np.sin(10*df['x1'])

Then the calculation is repeated for the ones with and without normalized y’s. The results and conclusions are obvious. Yes, you definitely want to do that.

Use neuronetwork(tensorflow) for regression 1

After discussion with Mengxi Wu, the poor prediction rate may come from the reason that the inputs are not normalized. Mengxi also points out that the initialization of weights may need to be related to the size of the input for each hidden neuron. In, it points out that the weights needs to be uniformly initialized between $$(-\frac{1}{\sqrt{d}},\frac{1}{\sqrt{d}})$$, where $$d$$ is the number of inputs to a given neuron.

The first step of modification is normalizing all inputs to its std equals 1.

std = df.std()
df = df/std

The training results is consistently above 96% and is obviously better than previous ones without normalization.

This comparison is repeated for hidden =[20,30,40,50,60,80,100], the prediction rate is more converged with larger numbers of hidden neurons. And the same plot is made without normalization.

Later on, we added weights initialization using Xaiver method. The results are also attached.

W1 = tf.get_variable("W1", shape=[FN,hidden],initializer=tf.contrib.layers.xavier_initializer())
W2 = tf.get_variable("W2", shape=[hidden,1],initializer=tf.contrib.layers.xavier_initializer())

Use neuronetwork(tensorflow) for regression 0

This post is not a tutorial, but rather a logbook of what we attempted.

The learning logbook starts with using neutonetwork to do regression.

The data is manually generate using a very simple formula.

$$y=3*x_0+sin(x_1)$$, initially, we do not add any noise term.

features = {}
for i in range(FN):
    features['x'+str(i)] = np.random.rand(2000)
df = pd.DataFrame(features)
df['y'] = 3*df['x0']+np.sin(10*df['x1'])

accuracy function is below

def accuracy(y,y0):
    return 1-np.sqrt(np.sum(np.square(y0-y)))/np.sqrt(np.sum(np.square(y0))).astype(np.float)

In the first attempt, a single intermediate layer neuronetwork is applied.

optimizer, prediction = None, None
def train(hidden,training_rate = 0.1,decay_rate = 0.96):
    optimizer, prediction = None, None
    graph = tf.Graph()
    with graph.as_default():
        x = tf.placeholder(tf.float32,[None, FN])
        y = tf.placeholder(tf.float32,[None, 1])

        W1 = tf.Variable(tf.truncated_normal([FN,hidden], stddev=0.001))
        b1 = tf.Variable(tf.zeros([hidden]))

        W2 = tf.Variable(tf.truncated_normal([hidden,1], stddev=0.001))
        b2 = tf.Variable(tf.zeros([1]))

        inter1 = tf.nn.relu(tf.matmul(x,W1) + b1)

        logits = tf.matmul(inter1,W2)+b2

        regularizers = tf.nn.l2_loss(W1) + tf.nn.l2_loss(W2)
        loss = tf.reduce_mean(tf.square(logits-y)) + 0*regularizers

        global_step = tf.Variable(0, trainable=False)
        learning_rate = tf.train.exponential_decay(training_rate, global_step,1000, decay_rate, staircase=True)
        optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss,global_step=global_step)
        prediction = logits

    ac = []
    st =[]
    r_previous = None
    with tf.Session(graph=graph) as session:
        for j in range(25000):

            feed_dict = {x:df.iloc[:1000,:-1].values,y:df.iloc[:1000,-1].values[:,None]}
            _, l, predict,r1,r2,lr =[optimizer,loss, prediction,W1,W2,learning_rate], feed_dict=feed_dict)
            if j%5000==0:
                acc = accuracy(predict.flatten(),df.iloc[:1000,-1].values[:1000])
                #print("step",j," is ",acc,l,lr,r1.shape)
                #print "train results (tmp):",predict.flatten()[:5]

                #if r_previous is not None:
                #    print "diff:",np.sqrt(np.sum(np.square(r_previous-r1)))
                r_previous = r1                    

        predict,r1,r2,lr =[prediction,W1,W2,learning_rate],feed_dict={x:df.iloc[1000:,:-1].values,y:df.iloc[1000:,-1].values[:,None]})
        #print "prediction:",accuracy(predict.flatten(),df.iloc[1000:,-1].values)
        #print "predicted results:",predict.flatten()[:10]
        return accuracy(predict.flatten(),df.iloc[1000:,-1].values)

It is observed that, with 60 hidden neurons and 25000 training steps, the prediction accuracy can be largely fluctuated according to the initialization value.

Fig 1. Prediction results with 60 hidden neurons, repeated 50 times.

Tensorflow strides

strides determines how much the window shifts by in each of the dimensions. The first (shift in the batch) and last (shift in the depth) stride needs to be 1

input size [batch, in_height, in_width, in_channels]
filter size [filter_height, filter_width, in_channels, out_channels]
stride size [batch_shift, width_shift, height_shift, color_shift]

Softmax vs softmax_cross_entropy_with_logits

cross entropy means $$ H =-\sum_i y_i’ log(y_i) $$$

loss=Softmax_cross_entropy_with_logits(logits, labels)

is equivalent to

normalized = Softmax(labels)
loss = cross_entropy(normalized, labels) ### you should know there is no such function in tensorflow as cross_entropy 

However, Softmax_cross_entropy_with_logits is somehow numerically more stable.