|Table of Contents|

 DMS Algorithm in the Application of the Map/Reduce Tasks Schedule

(PDF)

《哈尔滨理工大学学报》[ISSN:1007-2683/CN:23-1404/N]

Issue:
2019年01期
Page:
71-77
Research Field:
计算机与控制工程
Publishing date:

Info

Title:
 DMS Algorithm in the Application of the Map/Reduce Tasks Schedule

Author(s):
 PEI ShujunKONG DekaiMIAO Hui
 (School of Computer Science and Technology, Harbin University of Science and Technology, Harbin 150080, China)
Keywords:
 Keywords:cloud computing map/reduce tasks assign difference matrix
PACS:
TP319
DOI:
1015938/jjhust201901012
Abstract:
 Abstract:The whole efficiency of traditional task scheduling algorithms is low under the cloud environment, In order to improve the whole efficiency of the task scheduling, this article based on Map/Reduce presents a Difference Matrix Scheduling tasks schedule algorithm based on processing time Firstly, pretreatment of complex tasks, the complex tasks is converted to Directed Acyclic Graph figure, the tasks are topological sorted in an optimal manner according to the size of the task dependencies, and the work node is accordance with the sort to processing the complex tasks; Secondly, using the ratio of predictive time that node process tasks to node process capacity as a subtask in each node time quantitative modeling, then establish the task and the metric matrix of process time, according the Difference Matrix Scheduling to processing the matrix, and obtain the optimal scheme of task assignment. Finally, the experiment evaluates the Difference Matrix Scheduling ,fair scheduling algorithm, genetic algorithm in the task scheduling and resource utilization efficiency angles The results show that the algorithm can significantly improve the overall efficiency of complex task scheduling and make full use of the capacity of the compute nodes to improve the Map / Reduce scheduling efficiency

References:

 [1]黎建辉,沈志宏,孟小峰科学大数据管理:概念、技术与系统[J].计算机研究与发展,2016,54(2):235
[2]杨刚,杨凯大数据关键处理技术综述[J].计算机与数字工程,2016,44(4):694
[3]史恒亮云计算任务调度研究[D].南京:南京理工大学,2015
[4]杨志伟,郑烇,王嵩,等异构Spark集群下自适应任务调度策略[J].计算机工程,2016,42(1):31
[5]ZAREI Mohammad Hossein,SHIRSAVAR Milad Azizpour, YAZDANI NasserA QoSAware Task Allocation Model for Mobile Cloud Computing[C]//2nd International Conference on Web Research,Tehran, Iran,April 28,2016,Institute of Electrical and Electronics Engineers Inc,2016:43
[6]李德有,赵立波,解晨光Hadoop构建的银行海量数据存储系统研究[J].哈尔滨理工大学学报,2015,20(4):60
[7]杜江,张铮,张杰鑫,等MapReduce并行编程模型研究综述[J].计算机科学,2015,42(6A):537
[8]谢丽霞,严焱心云计算环境下的服务调度和资源调度研究[J].计算机应用研究,2015,32(2):528
[9]宋杰,刘雪冰,朱志良一种能效优化的MapReduce资源比模型[J].计算机学报,2015,38(1):59
[10]LIN J C,LEU F Y,CHEN Y PImpact of Map Reduce Policies on Job Completion Reliability and Job Energy Consumption[J].IEEE Transactions on Parallel & Distributed Systems,2015,26(5):1364
[11]张红,王晓明,曹洁,等Hadoop云平台MapReduce模型优化研究[J].计算机工程与应用,2016,52(22):22
[12]徐焕良,翟璐,薛卫,等Hadoop平台中MapReduce调度算法研究[J].计算机应用与软件,2015,32(5):1
[13]ARR Neto,ARR NetoA New Pruning Method for Extreme Learning Machines Via Genetic Algorithms[J].Elsevier Science Publishers B V, 2016, 44:101
[14]马月坤,刘鹏飞,张振友,等改进的FPGrowth算法及其分布式并行实现[J].哈尔滨理工大学学报,2016,21(2):20
[15]郑伟,马楠,一种改进的决策树后剪枝算法[J].计算机与数字工程,2015,6(43):960
[16]张新玲,颜秉珩Hadoop平台基准性能测试研究[J].软件导刊,2015,14(1):30
[17]冯兴杰,贺阳改进的Hadoop作业调度算法[J].计算机工程与应用,2016,53(12):85
[18]王波,张晓磊基于粒子群遗传算法的云计算任务调度研究[J].计算机工程与应用,2015,51(6):84
[19]胡艳华,唐新来基于改进遗传算法的云计算任务调度算法[J].计算机技术与发展,2016,10 (16):137
[20]万聪,王翠荣,王聪MapReduce模型中reduce阶段负载均衡分区算法研究[J].小型微型计算机系统,2015,36(2):240

Memo

Memo:
-
Last Update: 2019-03-26