Mnist Demo on EMR

本示例中的程序都可以在tf1.15或者tf2.0上运行

单机多卡模式: MirroredStragy

使用keras model，是tf2.x推荐运行的方式

wget https://easyrec.oss-cn-beijing.aliyuncs.com/data/mnist_demo/mnist.npz
hadoop fs -mkdir -p hdfs:///user/data/
hadoop fs -put mnist.npz hdfs:///user/data/

wget https://easyrec.oss-cn-beijing.aliyuncs.com/data/mnist_demo/mnist_mirrored.py -O mnist_mirrored.py
把strategy = tf.distribute.MirroredStrategy()
替换成strategy = tf.distribute.experimental.MultiWorkerMirroredStrategy()

el_submit  -t standalone -a mnist_train -f mnist_mirrored.py  -m local -wn 1 -wg 2  -wc 6  -wm 20000 -c python mnist_mirrored.py

• -wn: worker number，必须是1

• -wg: 2, 2GPUS

• -wc: CPU number

• -wm: cpu memory size in bytes, 20000 is 20G

多机多卡模式: MultiWorkerMirroredStrategy

wget https://easyrec.oss-cn-beijing.aliyuncs.com/data/mnist_demo/mnist.npz
hadoop fs -mkdir -p hdfs:///user/data/
hadoop fs -put mnist.npz hdfs:///user/data/
wget https://easyrec.oss-cn-beijing.aliyuncs.com/data/mnist_demo/mnist_mirrored.py -O mnist_mirrored.py

el_submit  -t tensorflow-worker -a mnist_train -f mnist_mirrored.py  -m local -wn 2 -wg 1  -wc 6  -wm 20000 -c python mnist_mirrored.py

• -wn: worker number，2

• -wg: 1, 1GPU, 可以 > 1

• -wc: CPU number

• -wm: cpu memory size in bytes, 20000 is 20G