Design of IfThenElse operator and layer

[wangkuiyi]

After talking with @zchen0211 and @Canpio , here is the very early draft of the idea of IfThenElse operator and layer:

A Simple Example

The Python Application Program

cost = paddle.layer.mse(
    paddel.layer.fc(
        paddle.layer.data(name="features"),
        output_size = 100),
    paddle.layer.data(name="label"))
parameters = paddle.train(cost, minist_reader(), {"features":0, "label":1})
print paddle.inf(cost, parameters, inference_input_reader())

FC Layer as a Python Function

def fc(input):
    output = paddle.framework.tensor()
    W = paddle.framework.tensor(label="parameter")
    b = paddle.framework.tensor(label="parameter")
    paddle.operator.fc(input, W, b, output)
    return output

FC Operator as a C++ Class

class FC : public framework:Operator {...};

The IfThenElse Layer and Operator

Another Example Using IfThenElse Layer

def a_network(input):
    A = paddle.layer.fc(input, parameter_name="A")
    B = paddle.layer.fc(input, parameter_name="B")
    return paddle.layer.if_then_else(
        paddle.layer.less_than(A, B),
        paddle.layer.fc(
            paddle.layer.fc(
                paddle.layer.fc
                paddle.layer.fc(A, output_size = 100))),
        paddle.layer.fc(
            paddle.layer.fc(
                paddle.layer.fc(B, output_size = 100))))

The IfThenElse Layer

def if_then_else(cond, left, right) :
    ret = paddle.framework.tensor()
    paddle.operator.if_then_else(cond, left, right, ret)
    return ret

The IfThenElseOp C++ Class

class IfThenElseOp : public framework::Operator {
 // Check that cond returns a minibatch of boolean values.
 // Check that left and right outputs are of the same size.
};

[Superjomn]

need a cond_op design.

cond_op get multiple inputs, and output a batch of bool?

maybe if_then_else is just a syntax wrapper of switch_op , cond_op(xxx) -> gather(xxx) -> switch_op(if_true_net, else_net) ?

gather_op is an operator to gather bools from cond_op and split the original input into different small batches of data, each small batch has the same cond_bool.

[zchen0211]

Tensorflow has a switch_op, and merge_op looks like this:

ret = tf.cond(cond, f_true, f_false)

cond: a bool scalar
f_true: a function
f_false: a function
when cond == true, ret = f_true(); ret=f_false(), otherwise.

In TF, it calls like this:

  x = tf.constant(2)
  y = tf.constant(5)
  def f1(): return tf.multiply(x, 17)
  def f2(): return tf.add(y, 23)
  r = tf.cond(tf.less(x, y), f1, f2)

r is set to f1().
Operations in f2 (e.g., tf.add) are not executed.

Since we are dynamic in Paddle, we need one more arguments input, the data to operate on

ret = paddle.cond(cond, f_true, f_false, input)

cond: a bool vector
f_true: a function
f_false: a function
when cond[i] == true, ret[i] = f_true(input[i]); ret[i]=f_false(input[i]), otherwise
Am I right?
@Superjom @wangkuiyi @emailweixu @Canpio

Our dynamic case

[Superjomn]

That works fine. cond_op should "tag" the inputs and split them into different batches, each batch has the items of different tags.

But I've no idea how to mark the inputs with the original orders, for example, in infer period, the switch_op will make inputs out of order, but the final outputs should be in the same order of the original inputs.

Do every input instance need an order id? by which the outputs are sorted?

@zchen0211

[emailweixu]

It's possible that the true and false branches need different input, so the interface should be like:
paddle.cond(cond, f_true, f_false, input, input_false=None)
when input_false is None, input is used for both f_true and f_false
when input_false is not None, "input" is used for f_true, "input_false" is used for f_false

[emailweixu]

For the actual user interface, it should be possible to make it such that the user does not need to specify input or input_false. The framework can figure out the input of f_true and f_false automatically.

[qingqing01]

Closed since IfElse operator had been done.