[1]裴树军,孔德凯,苗辉. DMS算法在Map/Reduce任务调度中的应用[J].哈尔滨理工大学学报,2019,(01):71-77.[doi:1015938/jjhust201901012]
 PEI Shu jun,KONG De kai,MIAO Hui. DMS Algorithm in the Application of the Map/Reduce Tasks Schedule[J].哈尔滨理工大学学报,2019,(01):71-77.[doi:1015938/jjhust201901012]
点击复制

 DMS算法在Map/Reduce任务调度中的应用
()
分享到:

《哈尔滨理工大学学报》[ISSN:1007-2683/CN:23-1404/N]

卷:
期数:
2019年01期
页码:
71-77
栏目:
计算机与控制工程
出版日期:
2019-08-06

文章信息/Info

Title:
 DMS Algorithm in the Application of the Map/Reduce Tasks Schedule

作者:
 裴树军孔德凯苗辉
(哈尔滨理工大学 计算机科学与技术学院,黑龙江 哈尔滨 150080)
Author(s):
 PEI ShujunKONG DekaiMIAO Hui
 (School of Computer Science and Technology, Harbin University of Science and Technology, Harbin 150080, China)
关键词:
 关键词:云计算Map/Reduce任务调度差值矩阵
Keywords:
 Keywords:cloud computing map/reduce tasks assign difference matrix
分类号:
TP319
DOI:
1015938/jjhust201901012
文献标志码:
A
摘要:
 摘要:云环境下传统的任务调度算法整体效率较低,为了提高任务调度的整体效率,在Map/Reduce基础上提出了一种基于处理时间的DMS任务调度算法。首先,对复杂任务进行预处理,将复杂任务转化为DAG图,依据任务依赖关系大小产生最佳拓扑排序,并依据排序结果将复杂任务交给work节点进行处理;其次,通过将节点处理任务的预测时间与节点处理能力的比值作为子任务在每个节点的处理“时间”进行量化建模,建立任务和处理时间的度量矩阵,依据DMS算法进行处理,从而获得任务分配最佳方案;最后,从任务调度效率与资源使用率的角度将DMS算法与公平调度算法、遗传算法行对比验证。实验结果表明,DMS算法能明显提高任务调度整体效率,充分利用各节点的计算能力提高了Map/Reduce的调度效率。
Abstract:
 Abstract:The whole efficiency of traditional task scheduling algorithms is low under the cloud environment, In order to improve the whole efficiency of the task scheduling, this article based on Map/Reduce presents a Difference Matrix Scheduling tasks schedule algorithm based on processing time Firstly, pretreatment of complex tasks, the complex tasks is converted to Directed Acyclic Graph figure, the tasks are topological sorted in an optimal manner according to the size of the task dependencies, and the work node is accordance with the sort to processing the complex tasks; Secondly, using the ratio of predictive time that node process tasks to node process capacity as a subtask in each node time quantitative modeling, then establish the task and the metric matrix of process time, according the Difference Matrix Scheduling to processing the matrix, and obtain the optimal scheme of task assignment. Finally, the experiment evaluates the Difference Matrix Scheduling ,fair scheduling algorithm, genetic algorithm in the task scheduling and resource utilization efficiency angles The results show that the algorithm can significantly improve the overall efficiency of complex task scheduling and make full use of the capacity of the compute nodes to improve the Map / Reduce scheduling efficiency

参考文献/References:

 [1]黎建辉,沈志宏,孟小峰科学大数据管理:概念、技术与系统[J].计算机研究与发展,2016,54(2):235
[2]杨刚,杨凯大数据关键处理技术综述[J].计算机与数字工程,2016,44(4):694
[3]史恒亮云计算任务调度研究[D].南京:南京理工大学,2015
[4]杨志伟,郑烇,王嵩,等异构Spark集群下自适应任务调度策略[J].计算机工程,2016,42(1):31
[5]ZAREI Mohammad Hossein,SHIRSAVAR Milad Azizpour, YAZDANI NasserA QoSAware Task Allocation Model for Mobile Cloud Computing[C]//2nd International Conference on Web Research,Tehran, Iran,April 28,2016,Institute of Electrical and Electronics Engineers Inc,2016:43
[6]李德有,赵立波,解晨光Hadoop构建的银行海量数据存储系统研究[J].哈尔滨理工大学学报,2015,20(4):60
[7]杜江,张铮,张杰鑫,等MapReduce并行编程模型研究综述[J].计算机科学,2015,42(6A):537
[8]谢丽霞,严焱心云计算环境下的服务调度和资源调度研究[J].计算机应用研究,2015,32(2):528
[9]宋杰,刘雪冰,朱志良一种能效优化的MapReduce资源比模型[J].计算机学报,2015,38(1):59
[10]LIN J C,LEU F Y,CHEN Y PImpact of Map Reduce Policies on Job Completion Reliability and Job Energy Consumption[J].IEEE Transactions on Parallel & Distributed Systems,2015,26(5):1364
[11]张红,王晓明,曹洁,等Hadoop云平台MapReduce模型优化研究[J].计算机工程与应用,2016,52(22):22
[12]徐焕良,翟璐,薛卫,等Hadoop平台中MapReduce调度算法研究[J].计算机应用与软件,2015,32(5):1
[13]ARR Neto,ARR NetoA New Pruning Method for Extreme Learning Machines Via Genetic Algorithms[J].Elsevier Science Publishers B V, 2016, 44:101
[14]马月坤,刘鹏飞,张振友,等改进的FPGrowth算法及其分布式并行实现[J].哈尔滨理工大学学报,2016,21(2):20
[15]郑伟,马楠,一种改进的决策树后剪枝算法[J].计算机与数字工程,2015,6(43):960
[16]张新玲,颜秉珩Hadoop平台基准性能测试研究[J].软件导刊,2015,14(1):30
[17]冯兴杰,贺阳改进的Hadoop作业调度算法[J].计算机工程与应用,2016,53(12):85
[18]王波,张晓磊基于粒子群遗传算法的云计算任务调度研究[J].计算机工程与应用,2015,51(6):84
[19]胡艳华,唐新来基于改进遗传算法的云计算任务调度算法[J].计算机技术与发展,2016,10 (16):137
[20]万聪,王翠荣,王聪MapReduce模型中reduce阶段负载均衡分区算法研究[J].小型微型计算机系统,2015,36(2):240

相似文献/References:

[1]孙永全,郭建英,陈洪科,等.AMSAA模型可靠性增长预测方法的改进[J].哈尔滨理工大学学报,2010,(05):49.
 SUN Yong-quan,GUO Jian-ying,CHEN Hong-ke,et al.An Improved Reliability Growth Prediction Algorithm Based on AMSAA Model[J].哈尔滨理工大学学报,2010,(01):49.
[2]滕志军,李晓霞,郑权龙,等.矿井巷道的MIMO信道几何模型及其信道容量分析[J].哈尔滨理工大学学报,2012,(02):14.
 TENG Zhi-jun,LI Xiao-xia,ZHENG Quan-long.Geometric Model for Mine MIMO Channels and Its Capacity Analysis[J].哈尔滨理工大学学报,2012,(01):14.
[3]李艳苹,张礼勇.新训练序列下的改进OFDM符号定时算法[J].哈尔滨理工大学学报,2012,(02):19.
 LI Yan-ping,ZHANG Li-yong.An Improved Algorithm of OFDM Symbol Timing Based on A New Training Sequence[J].哈尔滨理工大学学报,2012,(01):19.
[4]赵彦玲,车春雨,铉佳平,等.钢球全表面螺旋线展开机构运动特性分析[J].哈尔滨理工大学学报,2013,(01):37.
 ZHAO Yan-ling,CHE Chun-yu,XUAN Jia-ping,et al.[J].哈尔滨理工大学学报,2013,(01):37.
[5]李冬梅,卢旸,刘伟华,等.一类具有连续接种的自治SEIR传染病模型[J].哈尔滨理工大学学报,2013,(01):73.
 LI Dong-mei,LU Yang,LIU Wei-hua.[J].哈尔滨理工大学学报,2013,(01):73.
[6]华秀英,刘文德.奇Hamiltonian李超代数偶部的非负Z-齐次导子空间[J].哈尔滨理工大学学报,2013,(01):76.
 HUA Xiu-ying,LIU Wen-de.[J].哈尔滨理工大学学报,2013,(01):76.
[7]桂存兵,刘洋,何业军,等.基于LCC谐振电路阻抗匹配的光伏发电最大功率点跟踪[J].哈尔滨理工大学学报,2013,(01):90.
 GUI Cun-bing,LIU Yong,HE Ye-jun.[J].哈尔滨理工大学学报,2013,(01):90.
[8]翁凌,闫利文,夏乾善,等.PI/TiC@Al2O3复合薄膜的制备及其电性能研究[J].哈尔滨理工大学学报,2013,(02):25.
 WENG Ling,YAN Li-wen,XIA Qian-shan.[J].哈尔滨理工大学学报,2013,(01):25.
[9]姜彬,林爱琴,王松涛,等.高速铣刀安全性设计理论与方法[J].哈尔滨理工大学学报,2013,(02):63.
 JIANG Bin,LIN Ai-qin,WANG Song-tao,et al.[J].哈尔滨理工大学学报,2013,(01):63.
[10]李星纬,李晓东,张颖彧,等.EVOH 磺酸锂电池隔膜的制备及微观形貌[J].哈尔滨理工大学学报,2013,(05):18.
 LI Xing- wei,LI Xiao- dong,ZHANG Ying- yu,et al.The Preparation and Microcosmic Morphology oEVOH- SO Li Lithium Ion Battery Septum[J].哈尔滨理工大学学报,2013,(01):18.
[11]李成严,曹克翰,冯世祥,等. 不确定执行时间的云计算资源调度[J].哈尔滨理工大学学报,2019,(01):85.[doi:10.15938/j.jhust.2019.01.014]
 LI Cheng yan,CAO Ke han,FENG Shi xiang,et al. Resource Scheduling with Uncertain Execution Time in Cloud Computing[J].哈尔滨理工大学学报,2019,(01):85.[doi:10.15938/j.jhust.2019.01.014]

备注/Memo

备注/Memo:
 基金项目:国家自然科学基金(60572153,60972127)
更新日期/Last Update: 2019-03-26