Add client desgin by typhoonzero · Pull Request #17 · PaddlePaddle/PaddleCloud

typhoonzero · 2017-05-15T05:57:15Z

gongweibao · 2017-05-15T06:18:15Z

doc/client_design.md

+    `<dest>` is the path on the cloud. The form must be like `/pfs/$DATACENTER/home/$USER`. `$DATACENTER` is the from configuration where you setup at `~/.paddle/config`
+    - `paddlecloud pfs cp <local src> [<local src> ... ] <remote dest>`: Upload a file
+    - `paddlecloud pfs cp <remote src> [<remote src> ... ] <local dest>`: Download a file
+    - `paddlecloud pfs ls <remote dir>`: List files under `<remote dir>`.


简化的命令缺了一个
paddlecloud pfs rm <remote>

gongweibao · 2017-05-15T06:18:29Z

doc/client_design.md

+| /v1/submit | POST   | see [above](#client-commands) |
+| /v1/jobs   | GET    | see [above](#client-commands) |
+| /v1/quota  | GET    | see [above](#client-commands) |
+| /v1/pfs/cp | POST   | see [above](#client-commands) |


资源不应该是一个动词。

引用了Paddle下的design doc

Yancey0623 · 2017-05-15T05:58:11Z

doc/client_design.md

+
+# Goals:
+
+Developers using PaddlePadle Cloud can use this command-line client for conviniece of manage cloud Deep-Learning jobs, including:


command-line client => command-line interface

conviniece of manage cloud Deep-Learning jobs => managering cloud Deep-Learning jobs conveniently ?

这里有一个拼写错误convenience。command-line client的确是command-line client吧。这的确是个client而不是一个interface，interface要么是程序调用的要么是个服务端的API接口。

Yancey0623 · 2017-05-15T06:05:19Z

doc/client_design.md

+Developers using PaddlePadle Cloud can use this command-line client for conviniece of manage cloud Deep-Learning jobs, including:
+
+- Submitting a PaddlePaddle cluster training job
+- List jobs that is currently running.


is => are

Yancey0623 · 2017-05-15T06:06:00Z

doc/client_design.md

+
+- Submitting a PaddlePaddle cluster training job
+- List jobs that is currently running.
+- List all job history that have been submited.


List each job history that has been submitted?

需要列出所有的job不是一个一个列，所以是 all

Yancey0623 · 2017-05-15T06:18:07Z

doc/client_design.md

+    - `-m --memory`: Memory resource each trainer will use. Defaults to 1Gi.
+    - `-s --pservers`: Number of parameter servers. Defaults equal to `-p`
+    - `-u --pscpu`: Parameter server CPU resource. Defaults to 1.
+    - `-y --psmemory`: Parameter server momory resource. Defaults to 1Gi.


I think, it's better to show all suffixes, like

a plain integer using one of these suffixes: Ei, Pi, Ti, Gi, Mi, Ki

Yancey0623 · 2017-05-15T06:27:58Z

doc/client_design.md

+A Sample job package may contain files like below:
+
+```
+job_word_emb/


I don't think train.py and data directory in the same level is a good idea.
At the first step of paddlecloud submit, it will collecting the trainer package into a tar file, if data directory in the same level directory, we should tar -czvf --exculde ./data ... and the data directory maybe any name, so I think this one is better:

job_word_emb/ |-- module |-- trainer.py |-- dict1.pickle `-- my_topo.py `-- data |-- train |-- train.txt-000 ... `--test |-- test.txt-000 ...

submit的时候为什么不能上传数据呢，这样用户就不需要单独再上传一次数据了？

另外，使用tar原因如果为了压缩，可以直接使用gzip库，在读取文件的同时在内存中压缩并直接上传，没必要生成中间临时文件。

记录一下原因：每次submit的时候check一遍数据很慢，而且trainer.py的变化频率明显高于数据的变化频率

Yancey0623 · 2017-05-15T07:53:07Z

doc/client_design.md

+
+## Reference
+
+- `paddlecloud submit [options] [package path]`: submit job to PaddlePaddle Cloud


客户端是不是可以叫pcloud or pcloudctl，而且前面的介绍也可以统一使用这个名字了。

Yancey0623 · 2017-05-15T08:29:47Z

doc/client_design.md

+A Sample job package may contain files like below:
+
+```
+job_word_emb/


记录一下原因：每次submit的时候check一遍数据很慢，而且trainer.py的变化频率明显高于数据的变化频率

Yancey0623 · 2017-05-15T08:33:20Z

doc/client_design.md

+
+```bash
+# upload training data to cloud, which may be very large
+$ paddlecloud cp -r ./job_word_emb/data /pfs/datacenter1/home/user1/job_word_emb


上面写的是paddlecloud pfs cp ...，应该用什么呢？
另外paddlecloud的参数是不是应该按照paddlecloud [SUB_MODULE] [TYPE] [FLAGS]的格式统一呢？例如:
paddlecloud submit => paddlecloud job submit
paddlecloud logs => paddlecloud job logs
paddlecloud jobs => paddlecloud job ls

一般subcommand 没那么多层，参考：https://github.com/google/subcommands

都是paddlecloud [subcommand] [options...] [args...]只有两层

查看subcommand的help:
paddlecloud help [subcommand]

pfs的如果有可能，直接paddlecloud cp ...也会比较好。

对比kubectl的命令，也可以看到是以subcommand为设计原则的：

Basic Commands (Beginner): create Create a resource by filename or stdin expose Take a replication controller, service, deployment or pod and expose it as a new Kubernetes Service run Run a particular image on the cluster set Set specific features on objects Basic Commands (Intermediate): get Display one or many resources explain Documentation of resources edit Edit a resource on the server delete Delete resources by filenames, stdin, resources and names, or by resources and label selector Deploy Commands: rollout Manage a deployment rollout rolling-update Perform a rolling update of the given ReplicationController scale Set a new size for a Deployment, ReplicaSet, Replication Controller, or Job autoscale Auto-scale a Deployment, ReplicaSet, or ReplicationController ...

如果是subcommand的话，那动词看起来靠谱一些，不如pfs也统一改下？ @gongweibao

gongweibao

LGTM

Yancey0623

LGTM

Yancey0623

LGTM except one tiny comment!

Yancey0623 · 2017-05-15T09:22:11Z

doc/client_design.md

+# upload training data to cloud, which may be very large
+$ paddlecloud pfs cp -r ./job_word_emb/data /pfs/datacenter1/home/user1/job_word_emb
+# submit a v1 paddle training job
+$ paddlecloud submit ./job_word_emb -p 4 -c 2 -m 10Gi -t modules/train.py


paddlecloud submit ./job_word_emb -p 4 -c 2 -m 10Gi -t modules/train.py

=>
paddlecloud submit ./job_word_emb -p 4 -c 2 -m 10Gi -t train.py

* minor style change Signed-off-by: zhangzhengyuan <zhangzhengyuan0604@gmail.com> * style change Signed-off-by: zhangzhengyuan <zhangzhengyuan0604@gmail.com> * use stricter test in Reconcile Signed-off-by: zhangzhengyuan <zhangzhengyuan0604@gmail.com> * Revert "style change" This reverts commit ac60260dd328fb2ceb78d5fda67ff739f86f8469.

add client desgin

bb7ab5d

wangkuiyi previously approved these changes May 15, 2017

View reviewed changes

gongweibao requested changes May 15, 2017

View reviewed changes

Yancey0623 reviewed May 15, 2017

View reviewed changes

update

28e888e

typhoonzero dismissed wangkuiyi’s stale review via 28e888e May 15, 2017 08:08

add upload data first

eee0fbf

Yancey0623 reviewed May 15, 2017

View reviewed changes

gongweibao previously approved these changes May 15, 2017

View reviewed changes

update

f8934ce

typhoonzero dismissed gongweibao’s stale review via f8934ce May 15, 2017 08:47

Yancey0623 previously approved these changes May 15, 2017

View reviewed changes

add kill command, get command

9fa600d

typhoonzero dismissed Yancey0623’s stale review via 9fa600d May 15, 2017 09:15

Yancey0623 approved these changes May 15, 2017

View reviewed changes

typhoonzero merged commit 7932676 into PaddlePaddle:develop May 15, 2017

typhoonzero deleted the client_design branch November 2, 2017 10:04


		# Goals:

		Developers using PaddlePadle Cloud can use this command-line client for conviniece of manage cloud Deep-Learning jobs, including:


		## Reference

		- `paddlecloud submit [options] [package path]`: submit job to PaddlePaddle Cloud

Conversation

typhoonzero commented May 15, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gongweibao left a comment

Choose a reason for hiding this comment

Uh oh!

Yancey0623 left a comment

Choose a reason for hiding this comment

Uh oh!

Yancey0623 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants