Complete seq2seq for fluid by pkuyym · Pull Request #56 · dzhwinter/benchmark

pkuyym · 2018-01-15T14:50:29Z

Resolves #55
Resolves #22

[WIP] Add seq2seq model for fluid.

…fix-22

dzhwinter · 2018-01-16T06:32:44Z

fluid/machine_translation.py

+
+parser = argparse.ArgumentParser(description=__doc__)
+parser.add_argument(
+    "--word_vector_dim",


embedding_dim better?

dzhwinter · 2018-01-16T06:33:40Z

fluid/machine_translation.py

+import distutils.util
+
+import paddle.v2 as paddle
+import paddle.v2.fluid as fluid


Benchmark as a demo code, we'd like only have

+import paddle.v2 as paddle +import paddle.v2.fluid as fluid

just like tensorflow
import tensorflow as tf
nothing else.

dzhwinter · 2018-01-16T06:35:13Z

fluid/machine_translation.py

+    help="The dictionary capacity. Dictionaries of source sequence and "
+    "target dictionary have same capacity. (default: %(default)d)")
+parser.add_argument(
+    "--pass_number",


for unity, pass_num

dzhwinter · 2018-01-16T06:36:31Z

fluid/machine_translation.py

+    type=str,
+    default='train',
+    choices=['train', 'infer'],
+    help="Do training or inference. (default: %(default)s)")


infer_only
https://github.com/dzhwinter/benchmark/blob/master/fluid/resnet50.py#L60

Thanks, followed.

dzhwinter · 2018-01-16T06:41:27Z

fluid/machine_translation.py

+                   target_dict_dim,
+                   is_generating=False,
+                   beam_size=3,
+                   max_length=250):


leave max_length, beam_size default value to argparse.

dzhwinter · 2018-01-16T06:42:56Z

fluid/machine_translation.py

+    """Construct a seq2seq network."""
+    feeding_list = ["source_sequence", "target_sequence", "label_sequence"]
+
+    def bi_lstm_encoder(input_seq, size):


Here maybe we need a notation.
the lstm unit has 4 parameters, hidden, memory_cell, ...
so need to multiply by 4.

Add detailed comments.

dzhwinter · 2018-01-16T06:44:21Z

fluid/machine_translation.py

+                                             size=size * 4,
+                                             act='tanh')
+        forward, _ = fluid.layers.dynamic_lstm(
+            input=input_forward_proj, size=size * 4)


double check dynamic_lstm need 4. I roughly remember it has been done inside the lstm layer.

name a gate_size is a good idea.
https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/rnn.py#L27

double check dynamic_lstm need 4. I roughly remember it has been done inside the lstm layer.

https://github.com/PaddlePaddle/Paddle/blob/develop/python/paddle/v2/fluid/layers/nn.py#L231

name a gate_size is a good idea.

Agree.

dzhwinter · 2018-01-16T06:46:01Z

fluid/machine_translation.py

+    default=16,
+    help="The sequence number of a batch data. (default: %(default)d)")
+parser.add_argument(
+    "--dict_size",


This value is indicated by the dataset. should it be an argument?

Seems this is an argument:

https://github.com/PaddlePaddle/Paddle/blob/develop/python/paddle/v2/dataset/wmt14.py#L106

ranqiu92 · 2018-01-16T07:33:14Z

fluid/machine_translation.py

+    "--max_length",
+    type=int,
+    default=250,
+    help="The max length of sequence when doing generation. "


max -> maximum

ranqiu92 · 2018-01-16T07:40:50Z

fluid/machine_translation.py

+    "--batch_size",
+    type=int,
+    default=16,
+    help="The sequence number of a batch data. (default: %(default)d)")


of a mini-batch

ranqiu92 · 2018-01-16T07:41:02Z

fluid/machine_translation.py

+    "--encoder_size",
+    type=int,
+    default=512,
+    help="The size of encoder bi-rnn unit. (default: %(default)d)")


size -> dimension

Thinks both are ok, but size is shorter.

ranqiu92 · 2018-01-16T07:41:12Z

fluid/machine_translation.py

+    "--decoder_size",
+    type=int,
+    default=512,
+    help="The size of decoder rnn unit. (default: %(default)d)")


size -> dimension

Thinks both are ok, but size is shorter.

ranqiu92 · 2018-01-16T07:44:05Z

fluid/machine_translation.py

+    "--use_gpu",
+    type=distutils.util.strtobool,
+    default=True,
+    help="Whether use gpu. (default: %(default)d)")


Thanks, followed.

ranqiu92 · 2018-01-16T07:55:08Z

fluid/machine_translation.py

+
+    def lstm_decoder_with_attention(target_embedding, encoder_vec, encoder_proj,
+                                    decoder_boot, decoder_size):
+        def simple_attention(encoder_vec, encoder_proj, decoder_state):


The attention mechanism is wrong. Where is 'tanh' operation which appears in original formula?

Didn't catch your point, why tanh is necessary for attention? There are several kind of attention mechanisms. Please refer to https://github.com/PaddlePaddle/Paddle/blob/9bfa3013891cf3da832307894acff919d6705cee/python/paddle/trainer_config_helpers/networks.py#L1400

https://github.com/PaddlePaddle/Paddle/blob/9bfa3013891cf3da832307894acff919d6705cee/python/paddle/trainer_config_helpers/networks.py#L1473
Here, the mixed_layer performs tanh.
And for attention mechanism in Neural Machine Translation By Jointly Learning To Align and Translate, tanh is used. Is this what you want to realize?

Why do you think it's wrong to apply linear activation ?

To keep consistent, will apply tanh in next PR. Thanks.

dzhwinter

LGTM

dzhwinter · 2018-01-17T06:42:12Z

fluid/machine_translation.py

+
+            fetch_outs = exe.run(
+                inference_program,
+                feed=dict(zip(*[feeding_list, (src_seq, trg_seq, lbl_seq)])),


Please fix this issue.
even dict is better than *zip() function sugar.

pkuyym and others added 4 commits December 20, 2017 16:01

Add seq2seq model for fluid.

ce1ea0f

Merge pull request dzhwinter#25 from pkuyym/fix-22

5b8e29b

[WIP] Add seq2seq model for fluid.

Merge branch 'master' of https://github.com/dzhwinter/benchmark into …

d6e39f8

…fix-22

Complete seq2seq of fluid.

ba2ea17

pkuyym requested a review from dzhwinter January 15, 2018 14:50

pkuyym mentioned this pull request Jan 16, 2018

Implement attention based on RNN encoder-decoder PaddlePaddle/Paddle#6912

Closed

dzhwinter reviewed Jan 16, 2018

View reviewed changes

ranqiu92 reviewed Jan 16, 2018

View reviewed changes

Refine the script.

d5b7bb7

dzhwinter force-pushed the fix-55 branch from 5b51504 to d5b7bb7 Compare January 16, 2018 09:21

Not use peephole.

ac2b8c4

dzhwinter approved these changes Jan 17, 2018

View reviewed changes

pkuyym merged commit 315b20f into dzhwinter:master Jan 17, 2018

Comments

Conversation

pkuyym commented Jan 15, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ranqiu92 Jan 16, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pkuyym Jan 16, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dzhwinter left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

pkuyym commented Jan 15, 2018 •

edited

Loading

ranqiu92 Jan 16, 2018 •

edited

Loading

pkuyym Jan 16, 2018 •

edited

Loading