[Artificial Intelligence / TensorFlow] R-TensorFlow 예제 - Linear Regression

by Geol Choi | April 11, 2017

지난 포스팅에 이어 R-TensorFlow 세번째 예제로 Linear Regression을 구현하는 방법에 대하여 알아보기로 한다.

TensorFlow 라이브러리 로딩하기

지난 포스팅의 예제들과 마찬가지로 가장 먼저 할 일은, TensorFlow 라이브러리를 로딩하는 것이다. 이 외에도 Linear Regression을 시각화 하기 위해 plotly 라이브러리도 로딩하도록 한다:

R CODE:

# import library
if (! ("plotly" %in% rownames(installed.packages()))) { install.packages("plotly") }
library(plotly)

if (! ("tensorflow" %in% rownames(installed.packages()))) { install.packages("tensorflow") }
base::library(tensorflow)

파라미터 설정

학습률(Learning Rate; 최적화 문제에서는 Step Size라고도 함), Epochs(학습을 위한 반복계산 회수), Display Step(몇 번의 Epoch 주기마다 결과를 디스플레이를 할 지 결정하는 파라미터임)를 설정한다:

R CODE:

# Parameters
learning_rate <- 0.01
training_epochs <- 1000
display_step <- 50

학습 데이터 정의

Linear Regression에 대한 학습을 위한 데이터를 공급한다:

R CODE:

# Training Data
train_X <- base::c(3.3,4.4,5.5,6.71,6.93,4.168,9.779,6.182,7.59,2.167, 7.042,10.791,5.313,7.997,5.654,9.27,3.1)
train_Y <- base::c(1.7,2.76,2.09,3.19,1.694,1.573,3.366,2.596,2.53,1.221, 2.827,3.465,1.65,2.904,2.42,2.94,1.3)
n_samples <- base::length(train_X)

n_samples는 학습 데이터의 개수를 의미한다.

placeholder와 변수 정의

학습 연산 시 공급할 데이터에 대한 컨테이너(Container) 역할을 하는 placeholder와 변수를 정의한다:

R CODE:

# tf Graph Input
X <- tensorflow::tf$placeholder(tf$float32)
Y <- tensorflow::tf$placeholder(tf$float32)

# Set model weights
W <- tensorflow::tf$Variable(stats::runif(n = 1, min = -2, max = 2), name="weight")
b <- tensorflow::tf$Variable(stats::runif(n = 1, min = -2, max = 2), name="bias")

placeholder X와 Y의 데이터 타입은 float32(32-bit Single Precision Floating Point)로 지정하였으며, W와 b는 -2와 +2 사이의 Uniform Distribution에 의한 난수(Random Number) 1개를 발생시켰다.

Linear Model 정의

Linear Regression을 위한 모델, \( W^{T}x +b \)을 정의하는데, 목표는 W(Weight)와 b(Bias)를 결정하는 것이고 이를 위해 학습 데이터 X와 Y를 공급한다:

R CODE:

# Construct a linear model
pred <- tensorflow::tf$add(tensorflow::tf$multiply(X, W), b)

Cost Function 정의

Linear Regression 문제를 해결하기 위한 Cost Function(목적 함수; Objective Function)을 MSE(Mean-Square Error)로 하였다. 즉, 본 문제는 다음과 같은 최적화 문제를 해결하는 것과 같다:

\(\mathrm{min} \ f = \sum_{i=1}^{N}{\frac{(\bar{y}_i-y_i)^2}{2N}}\)

 

위의 식의 MSE에서 2로 나눈 이유는 제곱에 대한 미분을 할 때 계수(Coefficient)를 제거하기 위함이며 이로 인한 W와 b의 결과값에 대한 차이는 없다.

Cost Function을 정의하는 코드는 다음과 같다:

R CODE:

# Mean squared error
cost <- tensorflow::tf$reduce_sum(tensorflow::tf$pow(pred-Y, 2)) / (2*n_samples)

Optimizer 정의

Optimizer는 Newton 방법인 급강하법(Steepest Gradient Descent Method)을 사용하며, 매 Epoch에 대하여 다음과 같은 식으로 업데이트 된다:

\(W \leftarrow W - \alpha \nabla W \)

Optimizer를 정의하는 코드는 다음과 같다:

R CODE:

# Gradient descent
optimizer <- tensorflow::tf$train$GradientDescentOptimizer(learning_rate)$minimize(cost)

TensorFlow Session 정의 및 변수 초기화

다음과 같이 TensorFlow Session을 정의하며, 이 세션은 sess에 저장한다. 그리고나서 변수들을 초기화 한다:

R CODE:

# Launch the default graph.
sess <- tensorflow::tf$Session()

# Initializing the variables
sess$run(tf$global_variables_initializer())

데이터 학습 및 디스플레이 로그(Log) 저장

이제 데이터를 학습 시키도록 하는데, 매 Epoch 마다 train_X로부터 각 엘리먼트 별로 Optimizer를 실행한다:

R CODE:

# Fit all training data
for(epoch in 1:training_epochs) {
  for(index in 1:base::length(train_X)) {
    x <- train_X[index]; y <- train_Y[index];
    sess$run(optimizer, feed_dict = dict(X = x, Y = y))
  }
}

Epoch가 반복됨에 따라 Optimize가 되어 가는 과정을 디스플레이하기 위해 해당 결과값을 기록한다:

R CODE:

# Display logs per epoch step
if((epoch+1) %% display_step == 0) {
	sess$run(cost, feed_dict = dict(X = train_X, Y = train_Y))
	res <- base::sprintf("Epoch: %04d , cost = %.9f , W = %f, b = %f",
                     (epoch+1), sess$run(W), sess$run(W), sess$run(b))
	base::print(res)
}

위의 코드가 실행되면 다음과 유사한 결과가 출력될 것이다:

[1] "Epoch: 0050 , cost = 0.270439088 , W = 0.270439, b = 0.651506"
[1] "Epoch: 0100 , cost = 0.269199342 , W = 0.269199, b = 0.660424"
[1] "Epoch: 0150 , cost = 0.268033355 , W = 0.268033, b = 0.668812"
[1] "Epoch: 0200 , cost = 0.266936749 , W = 0.266937, b = 0.676701"
[1] "Epoch: 0250 , cost = 0.265905321 , W = 0.265905, b = 0.684121"
[1] "Epoch: 0300 , cost = 0.264935255 , W = 0.264935, b = 0.691100"
[1] "Epoch: 0350 , cost = 0.264022708 , W = 0.264023, b = 0.697664"
[1] "Epoch: 0400 , cost = 0.263164729 , W = 0.263165, b = 0.703837"
[1] "Epoch: 0450 , cost = 0.262357503 , W = 0.262358, b = 0.709644"
[1] "Epoch: 0500 , cost = 0.261598349 , W = 0.261598, b = 0.715105"
[1] "Epoch: 0550 , cost = 0.260884255 , W = 0.260884, b = 0.720242"
[1] "Epoch: 0600 , cost = 0.260212690 , W = 0.260213, b = 0.725073"
[1] "Epoch: 0650 , cost = 0.259581059 , W = 0.259581, b = 0.729617"
[1] "Epoch: 0700 , cost = 0.258986861 , W = 0.258987, b = 0.733892"
[1] "Epoch: 0750 , cost = 0.258428127 , W = 0.258428, b = 0.737911"
[1] "Epoch: 0800 , cost = 0.257902652 , W = 0.257903, b = 0.741692"
[1] "Epoch: 0850 , cost = 0.257408261 , W = 0.257408, b = 0.745248"
[1] "Epoch: 0900 , cost = 0.256943405 , W = 0.256943, b = 0.748592"
[1] "Epoch: 0950 , cost = 0.256506115 , W = 0.256506, b = 0.751739"
[1] "Epoch: 1000 , cost = 0.256094903 , W = 0.256095, b = 0.754697"

학습 결과값 및 그래프 출력

학습된 최종 결과를 확인하도록 한다. 최종 결과에는 학습 결과에 대한 Cost Function 값, W, b의 값이다:

R CODE:

base::print("Optimization Finished!")
training_cost <- sess$run(cost, feed_dict = dict(X = train_X, Y = train_Y))

base::print(base::sprintf("Training cost = %f, W = %f, b = %f",
                          training_cost,
                          sess$run(W),
                          sess$run(b)))

필자의 컴퓨터로 실행한 결과는 다음과 같았다:

[1] "Training cost = 0.077116, W = 0.256087, b = 0.754754"

그러나, 숫자로 표기되니 결과(Fitted Line)이 원래의 데이터와 얼마나 잘 맞는지 알기가 어려우므로 plotly 패키지 라이브러리를 이용하여 이를 시각화해 본다:

R CODE:

# Graphic display for train data
df1 <- base::data.frame(base::matrix(ncol = 3, nrow = base::length(train_X)))
base::names(df1) <- base::c("train_X", "train_Y", "fitted")
df1$train_X <- train_X
df1$train_Y <- train_Y
df1$fitted <- sess$run(W) * train_X + sess$run(b)

p <- plotly::plot_ly()
p <- p %>% plotly::add_trace(data = df1,
                             x = ~train_X,
                             y = ~train_Y,
                             type="scatter",
                             mode = "markers",name = "Train Data")
p <- p %>% plotly::add_trace(data = df1,
                             x = ~train_X,
                             y = ~fitted,
                             type="scatter",
                             mode = "lines",
                             name = "Fitted Line")
p <- p %>% plotly::layout(title = 'Linear Regression for Train Data',
                          yaxis = base::list(title = "Y", zeroline = FALSE),
                          xaxis = base::list(title = "X", zeroline = FALSE))

base::print(p) # print results

테스트 데이터를 통한 평가

이제 테스트 데이터를 공급하고 이들을 통해 손실(Loss)과 MSE를 계산한다:

R CODE:

# Testing example, as requested
test_X <- base::c(6.83, 4.668, 8.9, 7.91, 5.7, 8.7, 3.1, 2.1)
test_Y <- base::c(1.84, 2.273, 3.2, 2.831, 2.92, 3.24, 1.35, 1.03)

base::print("Testing... (Mean square loss Comparison)")
testing_cost <- sess$run(tf$reduce_sum(tf$pow(pred - Y, 2)) / (2 * base::length(test_X)),
                         feed_dict = dict(X = test_X, Y = test_Y))  # same function as cost above

base::print(base::sprintf("Testing cost = %f", testing_cost))
base::print(base::sprintf("Absolute mean square loss difference: %f", base::abs(training_cost - testing_cost)))

위의 코드 실행결과는 다음과 같았다:

[1] "Epoch: 0050 , cost = 0.381595880 , W = 0.381596, b = -0.148148"
[1] "Epoch: 0100 , cost = 0.373745024 , W = 0.373745, b = -0.091670"
[1] "Epoch: 0150 , cost = 0.366361171 , W = 0.366361, b = -0.038551"
[1] "Epoch: 0200 , cost = 0.359416485 , W = 0.359416, b = 0.011409"
[1] "Epoch: 0250 , cost = 0.352884799 , W = 0.352885, b = 0.058397"
[1] "Epoch: 0300 , cost = 0.346741557 , W = 0.346742, b = 0.102591"
[1] "Epoch: 0350 , cost = 0.340963721 , W = 0.340964, b = 0.144157"
[1] "Epoch: 0400 , cost = 0.335529476 , W = 0.335529, b = 0.183250"
[1] "Epoch: 0450 , cost = 0.330418438 , W = 0.330418, b = 0.220018"
[1] "Epoch: 0500 , cost = 0.325611353 , W = 0.325611, b = 0.254600"
[1] "Epoch: 0550 , cost = 0.321090162 , W = 0.321090, b = 0.287125"
[1] "Epoch: 0600 , cost = 0.316837847 , W = 0.316838, b = 0.317716"
[1] "Epoch: 0650 , cost = 0.312838763 , W = 0.312839, b = 0.346486"
[1] "Epoch: 0700 , cost = 0.309077144 , W = 0.309077, b = 0.373546"
[1] "Epoch: 0750 , cost = 0.305539340 , W = 0.305539, b = 0.398997"
[1] "Epoch: 0800 , cost = 0.302212059 , W = 0.302212, b = 0.422933"
[1] "Epoch: 0850 , cost = 0.299082607 , W = 0.299083, b = 0.445446"
[1] "Epoch: 0900 , cost = 0.296139181 , W = 0.296139, b = 0.466621"
[1] "Epoch: 0950 , cost = 0.293370932 , W = 0.293371, b = 0.486536"
[1] "Epoch: 1000 , cost = 0.290767282 , W = 0.290767, b = 0.505266"
[1] "Optimization Finished!"
[1] "Training cost = 0.082352, W = 0.290717, b = 0.505629"
[1] "Testing... (Mean square loss Comparison)"
[1] "Testing cost = 0.076923"
[1] "Absolute mean square loss difference: 0.005429"

마지막으로 테스트 데이터에 대한 결과를 그래프로 출력해 보도록 한다:

R CODE:

# Graphic display for test data
df2 <- base::data.frame(base::matrix(ncol = 2, nrow = base::length(test_X)))
base::names(df2) <- base::c("test_X", "test_Y")
df2$test_X <- test_X
df2$test_Y <- test_Y

p <- plotly::plot_ly()
p <- p %>% plotly::add_trace(data = df2,
                             x = ~test_X,
                             y = ~test_Y,
                             type="scatter",
                             mode = "markers",name = "Test Data")
p <- p %>% plotly::add_trace(data = df1,
                             x = ~train_X,
                             y = ~fitted,
                             type="scatter",
                             mode = "lines",
                             name = "Fitted Line")
p <- p %>% plotly::layout(title = 'Linear Regression for Test Data',
                          yaxis = base::list(title = "Y", zeroline = FALSE),
                          xaxis = base::list(title = "X", zeroline = FALSE))

base::print(p) # print results

전체 소스 코드

전체 소스 코드를 수록하고 본 포스팅을 마치고자 한다.

linear_regression.R

#############################################################################
# linear regression example using TensorFlow library.
 
# Author: Aymeric Damien
# Translated to R: Geol Choi, phD.(cinema4dr12@gmail.com)
# Date: April 10, 2017
#############################################################################
base::rm(list = ls())
base::gc()
 
# import libraries
if (! ("plotly" %in% rownames(installed.packages()))) { utils::install.packages("plotly") }
library(plotly)
 
if (! ("tensorflow" %in% rownames(installed.packages()))) { utils::install.packages("tensorflow") }
base::library(tensorflow)
 
# Parameters
learning_rate <- 0.01
training_epochs <- 1000
display_step <- 50
 
# Training Data
train_X <- base::c(3.3,4.4,5.5,6.71,6.93,4.168,9.779,6.182,7.59,2.167, 7.042,10.791,5.313,7.997,5.654,9.27,3.1)
train_Y <- base::c(1.7,2.76,2.09,3.19,1.694,1.573,3.366,2.596,2.53,1.221, 2.827,3.465,1.65,2.904,2.42,2.94,1.3)
n_samples <- base::length(train_X)
 
# tf Graph Input
X <- tensorflow::tf$placeholder(tf$float32)
Y <- tensorflow::tf$placeholder(tf$float32)
 
# Set model weights
W <- tensorflow::tf$Variable(stats::runif(n = 1, min = -2, max = 2), name="weight")
b <- tensorflow::tf$Variable(stats::runif(n = 1, min = -2, max = 2), name="bias")
 
# Construct a linear model
pred <- tensorflow::tf$add(tensorflow::tf$multiply(X, W), b)
 
# Mean squared error
cost <- tensorflow::tf$reduce_sum(tensorflow::tf$pow(pred-Y, 2)) / (2*n_samples)
 
# Gradient descent
optimizer <- tensorflow::tf$train$GradientDescentOptimizer(learning_rate)$minimize(cost)
 
# Launch the default graph.
sess <- tensorflow::tf$Session()
 
# Initializing the variables
sess$run(tf$global_variables_initializer())
 
# Fit all training data
for(epoch in 1:training_epochs) {
  for(index in 1:base::length(train_X)) {
    x <- train_X[index]; y <- train_Y[index];
    sess$run(optimizer, feed_dict = dict(X = x, Y = y))
  }
  
  # Display logs per epoch step
  if((epoch+1) %% display_step == 0) {
    sess$run(cost, feed_dict = dict(X = train_X, Y = train_Y))
    res <- base::sprintf("Epoch: %04d , cost = %.9f , W = %f, b = %f",
                         (epoch+1), sess$run(W), sess$run(W), sess$run(b))
    base::print(res)
  }
}
 
base::print("Optimization Finished!")
training_cost <- sess$run(cost, feed_dict = dict(X = train_X, Y = train_Y))
 
base::print(base::sprintf("Training cost = %f, W = %f, b = %f",
                          training_cost,
                          sess$run(W),
                          sess$run(b)))
 
# Graphic display for train data
df1 <- base::data.frame(base::matrix(ncol = 3, nrow = base::length(train_X)))
base::names(df1) <- base::c("train_X", "train_Y", "fitted")
df1$train_X <- train_X
df1$train_Y <- train_Y
df1$fitted <- sess$run(W) * train_X + sess$run(b)
 
p <- plotly::plot_ly()
p <- p %>% plotly::add_trace(data = df1,
                             x = ~train_X,
                             y = ~train_Y,
                             type="scatter",
                             mode = "markers",name = "Train Data")
p <- p %>% plotly::add_trace(data = df1,
                             x = ~train_X,
                             y = ~fitted,
                             type="scatter",
                             mode = "lines",
                             name = "Fitted Line")
p <- p %>% plotly::layout(title = 'Linear Regression for Train Data',
                          yaxis = base::list(title = "Y", zeroline = FALSE),
                          xaxis = base::list(title = "X", zeroline = FALSE))
 
base::print(p) # print results
 
# Testing example, as requested
test_X <- base::c(6.83, 4.668, 8.9, 7.91, 5.7, 8.7, 3.1, 2.1)
test_Y <- base::c(1.84, 2.273, 3.2, 2.831, 2.92, 3.24, 1.35, 1.03)
 
base::print("Testing... (Mean square loss Comparison)")
testing_cost <- sess$run(tf$reduce_sum(tf$pow(pred - Y, 2)) / (2 * base::length(test_X)),
                         feed_dict = dict(X = test_X, Y = test_Y))  # same function as cost above
 
base::print(base::sprintf("Testing cost = %f", testing_cost))
base::print(base::sprintf("Absolute mean square loss difference: %f", base::abs(training_cost - testing_cost)))
 
# Graphic display for test data
df2 <- base::data.frame(base::matrix(ncol = 2, nrow = base::length(test_X)))
base::names(df2) <- base::c("test_X", "test_Y")
df2$test_X <- test_X
df2$test_Y <- test_Y
 
p <- plotly::plot_ly()
p <- p %>% plotly::add_trace(data = df2,
                             x = ~test_X,
                             y = ~test_Y,
                             type="scatter",
                             mode = "markers",name = "Test Data")
p <- p %>% plotly::add_trace(data = df1,
                             x = ~train_X,
                             y = ~fitted,
                             type="scatter",
                             mode = "lines",
                             name = "Fitted Line")
p <- p %>% plotly::layout(title = 'Linear Regression for Test Data',
                          yaxis = base::list(title = "Y", zeroline = FALSE),
                          xaxis = base::list(title = "X", zeroline = FALSE))
 
base::print(p) # print results

Python-TensorFlow 코드

전참고로 위의 R코드의 Python-TensorFlow 코드는 다음과 같다:

LinearRegression.py

'''
A linear regression learning algorithm example using TensorFlow library.

Author: Aymeric Damien
Project: https://github.com/aymericdamien/TensorFlow-Examples/
'''
 
from __future__ import print_function
 
import tensorflow as tf
import numpy
import matplotlib.pyplot as plt
rng = numpy.random
 
# Parameters
learning_rate = 0.01
training_epochs = 1000
display_step = 50
 
# Training Data
train_X = numpy.asarray([3.3,4.4,5.5,6.71,6.93,4.168,9.779,6.182,7.59,2.167,
                         7.042,10.791,5.313,7.997,5.654,9.27,3.1])
train_Y = numpy.asarray([1.7,2.76,2.09,3.19,1.694,1.573,3.366,2.596,2.53,1.221,
                         2.827,3.465,1.65,2.904,2.42,2.94,1.3])
n_samples = train_X.shape[0]
 
# tf Graph Input
X = tf.placeholder("float")
Y = tf.placeholder("float")
 
# Set model weights
W = tf.Variable(rng.randn(), name="weight")
b = tf.Variable(rng.randn(), name="bias")
 
# Construct a linear model
pred = tf.add(tf.multiply(X, W), b)
 
# Mean squared error
cost = tf.reduce_sum(tf.pow(pred-Y, 2))/(2*n_samples)
 
# Gradient descent
optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)
 
# Initializing the variables
init = tf.global_variables_initializer()
 
# Launch the graph
with tf.Session() as sess:
    sess.run(init)
 
    # Fit all training data
    for epoch in range(training_epochs):
        for (x, y) in zip(train_X, train_Y):
            sess.run(optimizer, feed_dict={X: x, Y: y})
 
        # Display logs per epoch step
        if (epoch+1) % display_step == 0:
            c = sess.run(cost, feed_dict={X: train_X, Y:train_Y})
            print("Epoch:", '%04d' % (epoch+1), "cost=", "{:.9f}".format(c), \
                "W=", sess.run(W), "b=", sess.run(b))
 
    print("Optimization Finished!")
    training_cost = sess.run(cost, feed_dict={X: train_X, Y: train_Y})
    print("Training cost=", training_cost, "W=", sess.run(W), "b=", sess.run(b), '\n')
 
    # Graphic display
    plt.plot(train_X, train_Y, 'ro', label='Original data')
    plt.plot(train_X, sess.run(W) * train_X + sess.run(b), label='Fitted line')
    plt.legend()
    plt.show()
 
    # Testing example, as requested (Issue #2)
    test_X = numpy.asarray([6.83, 4.668, 8.9, 7.91, 5.7, 8.7, 3.1, 2.1])
    test_Y = numpy.asarray([1.84, 2.273, 3.2, 2.831, 2.92, 3.24, 1.35, 1.03])
 
    print("Testing... (Mean square loss Comparison)")
    testing_cost = sess.run(
        tf.reduce_sum(tf.pow(pred - Y, 2)) / (2 * test_X.shape[0]),
        feed_dict={X: test_X, Y: test_Y})  # same function as cost above
    print("Testing cost=", testing_cost)
    print("Absolute mean square loss difference:", abs(
        training_cost - testing_cost))
 
    plt.plot(test_X, test_Y, 'bo', label='Testing data')
    plt.plot(train_X, sess.run(W) * train_X + sess.run(b), label='Fitted line')
    plt.legend()
    plt.show()

 

실행결과는 다음과 같다.

Epoch: 0050 cost= 0.366798908 W= 0.550465 b= -1.36298
Epoch: 0100 cost= 0.333326221 W= 0.532571 b= -1.23425
Epoch: 0150 cost= 0.303718448 W= 0.515741 b= -1.11318
Epoch: 0200 cost= 0.277529180 W= 0.499912 b= -0.999306
Epoch: 0250 cost= 0.254363596 W= 0.485024 b= -0.892206
Epoch: 0300 cost= 0.233873099 W= 0.471022 b= -0.791476
Epoch: 0350 cost= 0.215748549 W= 0.457853 b= -0.696736
Epoch: 0400 cost= 0.199716866 W= 0.445467 b= -0.607631
Epoch: 0450 cost= 0.185536489 W= 0.433817 b= -0.523826
Epoch: 0500 cost= 0.172993660 W= 0.422861 b= -0.445005
Epoch: 0550 cost= 0.161899269 W= 0.412556 b= -0.370871
Epoch: 0600 cost= 0.152086198 W= 0.402864 b= -0.301147
Epoch: 0650 cost= 0.143406436 W= 0.393748 b= -0.235569
Epoch: 0700 cost= 0.135729223 W= 0.385174 b= -0.173892
Epoch: 0750 cost= 0.128938764 W= 0.377111 b= -0.115882
Epoch: 0800 cost= 0.122932658 W= 0.369527 b= -0.0613231
Epoch: 0850 cost= 0.117620409 W= 0.362394 b= -0.0100088
Epoch: 0900 cost= 0.112921841 W= 0.355685 b= 0.0382534
Epoch: 0950 cost= 0.108766086 W= 0.349375 b= 0.0836452
Epoch: 1000 cost= 0.105090529 W= 0.343441 b= 0.126337
Optimization Finished!
Training cost= 0.105091 W= 0.343441 b= 0.126337
Testing... (Mean square loss Comparison)
Testing cost= 0.0917893
Absolute mean square loss difference: 0.0133012